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I. INTRODUCTION 


There are many definitions available for the term "human 
factors". Simply stated, the major concern of all people in- 
volved with human factors is that of providing equipment and 
systems which people can use. Included in the various facets 
of human factors are design, test and evaluation. 

Technology has advanced rapidly during the last decade. 
The development of the people who must make use of these 
"state-of-the-art" technological wonders has not kept abreast 
of the rapid growth in this field. For this reason, there 
has been increased attention to the human factors aspects of 
the new systems. Design engineers and evaluators must be 
cognizant of this fact, so that rather than designing equip- 
ment for fellow technical wizards' use, they must take into 
consideration the attributes and capabilities of the ulti- 
mate user. [Ref. 1: pp. 1-27] Referred to in U. S. Navy 
femme, the requirement 1S to make it "sailor-proof". Be- 
Gause the majority of sailors or non-prior service enlisted 
personnel as measured by the Armed Service Vocational Bat- 
tery (ASVAB) (Table 1) fall into the lower mental categories 
[Ref. 2: p. 281], this requirement is of major concern when 
implementing new systems or equipment for their operation. 

Failure of deployed systems are often caused by human 


induced errors. There are indications that the percentage 
of failures due to human ineptitude or poor human 
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reliability! may be quite high. The increasingly compli- 
cated nature of modern military systems together with | 
shortages of qualified military personnel suggests that 
human-induced errors both in operation and maintenance of 
Systems will increase unless more attention is given to 
this problem in the design and development phases of the 
acquisition process. [Ref. 1: p. 27] 


Table l 


Mental Category of Non-Prior Service 
Enlisted Accessions for 1978 


Mental Population Marine Air 
Category Percentile DOD Army Navy Corps_ Force 
I + II 65>) 1 UU 34 26 #38 29 45 
iI 31 -. 64 ail Were 60 68 aS 


IV 10 - 30 3 MIRE 5 4 0 


The Department of Defense [Ref. 1: p. 34] is aware of 
this problem and concurs that effective action must be taken 
Memcoarrect it. 

Existing military instructions and directives [Ref. 4] 
State that human factors engineering must be integrated into 
the standard test and evaluation plans of all military sys- 


tems. Command level documents [Ref. 5] include statements 


THuman reliability as used here is defined as the probabil- 
aweniat Human error by either the operator or the maintainer 
will not cause a system failure or malfunction. The concept 
of system performance employed by the HFE Self-Paced Course 
1s not concerned with human error that does not cause system 
failure. The evaluator concentrates on overall system per- 
formance. "The human is an integral part of a system and 
therefore his/her performance becomes a system component for 
evaluation. [Ref. 3: pp. 3-14] 
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which say that human factors aspects of equipments and sys- 
tems will be assessed. All too frequently, however, test 
directors design the evaluation plan without human Raedens 
tests, or with human factors testing scheduled on ‘ "not-to- 
materrere basis” or without Comsidering the basic "...char- 
acteristics of personnel who will operate and maintain the 
system when it is eventually deployed, i.e., muscular 
strength and coordination, body dimensions, perceptions and 
judgment, sensory capacities, native skills and capacity to 
learn new skills, optimum workload, basic requirements “for 
comfort, safety, and freedom from environmental stress." 
feet. Ls p. 29] 

An additional problem arises when the evaluators become 
involved too late in the development and- acquisition cycle-- 
the "die have been cast"; the changes required are too ex- 
pensive in terms of time and/or ieitarae Tiius “test directors 
Seemeercea into the world of trade-offs--instead of receiving 
the best,.the user will get whatever the least costly compro- 
imecceaad tO Offer Or whatever “hurt™ the least. This often 
occurs because of the evaluator's inexperience with the design, 
test and evaluation processes and/or lack of guidance, train- 
ing, and education in the field of human factors. [Ref 1: p. 
POF ZL; °S1} 

Perrouswerements of the Department of Defé€nse and civilian 
authors have attempted to develop documents which provide hu- 


man factors engineering test and evaluation guidelines. Some 
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of these guidelines tell the planner when the evaluator 
should first becomé involved, others provide evaluation check- 
lists and criteria against which to measure. A variety of 
documents even tell how to nouns the tests. From discus- 
sions held with various members in the test and evaluation 
field, indications are that most of these existing references 
lie in sad disuse. They add that for all the expertise in- 
volved in their development, no one has to date, provided 
the one link that would put these documents into high demand: 
basic training and education in human factors. Valuable 
documents such as Human Factors Test and Evaluation Manual 
(HFTEMAN), Human Factors Engineering Data Guide for Evalua- 
tion (HEDGE), and Mission Operability Assessment Technique 
(MOAT), to name just a -few, require a basic knowledge of 
human factors for their effective use. Consistent with hu- 
man nature, it ees, to put off, ignore, or’ down play the 
importance of those things which are not easily understood. 
Having recognized the need for basic training in human 
factors, the question then becomes one of implementation. 
thesmeans of instruction must take into account cost effec- 
tiveness, and instruction standardized so that all intended 
users may benefit. During the 1970's and 1980's, the use of 
the Personalized System of Instruction (PSI) has gained in 
popularity and credibility [Ref 6: pp. 1-9] because of its 
economical feasibility [Ref 6: pp. 11-16] and attention to 


different individual learning styles [Ref. 6: pp. 73-75]. 
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Critical aspects of the previously heretical concepts of PSI 
hdve been undergoing intensive study [Ref. 6: pp. 1-9; Ref. 7: 
pp. 587-592] with proven postemme results. The major success 
or failure of PSI is greatly dependent upon its design con- 
Siderations. The subject matter expert usually knows what 
content must be included, but often times lacks the famili- 
arity and understanding necessary for PSI development to give 
all students an equal opportunity to master the content. 
Frequently, the problem lies in its method of presentation ~ 
and delivery. [Refs 8: pp. 40-43] Adherence to the basic 
requirements for a PSI design and development can produce an 
extremely effective course [Ref. 9: pp. 165-170]. 

Progressing with the advances of modern education and 
seeking cost effectiveness in delivering instruction, the 
HFE Self-Paced Course designers elected to develop a PSI for 
ee with its small and specialized enrollment. The U. S. © 
Meenas formally recognized procedures and agencies for PSI 
development, design, implementation and evaluation [Refs. 10, 
ie). 

It 1s the purpose of this thesis to investigate a human 
factors engineering self-paced course which is anticipated 
to provide the missing link--education and training in basic 
human factors. Performance of this investigation will be 


accomplished in three phases: 
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1. Development of an evaluation plan for a self-paced 
course in human factors in accordance alas CNET and CNTECHTRA 
faarectives. 

Z2. Implementation Of the evaluation plan in a realistic 
environment, and ; 

we evaluation of the Human Factors Engineering Self- 
Paced Course. 

A description of a preliminary draft of the Human Factors 
Engineering Self-Paced Course is provided in Chapter II. The 
evaluation plan used for the erecetiveness assessment of the 
course is discussed in Chapter III. Methodology and-results 
are addressed in Chapter IV. Chapter V presents this author's 
conclusions and recommendations. A final summary is given 


foenapter VI. 
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II. THE HUMAN FACTORS ENGINEERING (HFE) SELF-PACED COURSE 


The Human Factors Engineering Self-Paced Course is clas- 
Sified as a Self-Teaching Exportable Package (STEP) by . 
NAVEDTRA 106A. As such, this course is intended to be self- 
Supporting with minimal student contact required with the 
distributing command. 

Once the course is implemented into the system, any mili- 
tary member may order the course materials Gaerne SpOnSOT - 
ing eeiiiand or through standard military supply channels. 
Upon receipt of the course Parone) s, the student may proceed 
With the lessons, working at home or during free time while 
"on the job". A recommended schedule is provided to each 
Student as a guideline for meaningful and timely completion 
of the course materials. Thus the participant proceeds through 
the lessons on a self-paced basis while continuing to perform 
duties and tasks at the parent command. 

This self-paced course has two formats: (1) a set of 
written manuals, and (2) a computer-based EURO OAPI planned to 
be available on the Advanced Research Project Agency (ARPA) 
network on a time-share basis. The total cost of either pro- 
gram 1S estimated to be considerably less in dollars (travel 
and per diem), man-hours required for training attendance, 
and man-hours lost from performing on-the-job tasks and 
duties while attending training than sending personnel to off- 


Site human factors training facilities. This evaluation was 


20 





conducted using the written manuals because the computer 
version was not accessable at the time.° 

Because the Teen mduertalsware sso extensame, they are 
not provided as part of this thesis. However, excerpts and 
Summarizations taken from the course materials covering the 
course purpose, design, description, learning objectives and 


content are provided in the remainder of this section. 


pee PURPOSE 

This self-paced course was developed to provide HFE 
practioners with an awareness of its facts, principles and 
issues. The course designers have intended that upon comple- 
tion of their course of study, each participant should "think. 
about human factors whenever they are doing their job". 
Therefore, its purpose is to bridge the gap between PDepart- 
ment of Defense (DOD) directives and instructions on HFE de- 
Sign, test and evaluation and the knowledge level of the 


novice HFE evaluator. 


B. COURSE DESIGN AND STRUCTURE 
Preeire sel: -Paced Course is a STEP [{Ref. 10: p.140], 


consisting of a student supplement, 40 lessons, and various 


the computer version has been written but time and fi- 
nancial constraints made it impossible to evaluate. Portable 
Compucer terminals were required for distribution at test 
Sites and these were not available. The program had also not 
been entered on to the ARPA network for systemwide access. 
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military standards (MIL-STDS), handbooks (HDBKs) and docu- 
ments. The Course is intended to provide the student with 
aii Same sort of instruction that would be found if she/he 
were the only student, of an instructor. The written manuals 
consist of eight lesson books, each containing five lessons. 
The student supplement is to be used in concert with either 
the written or computer generated format. 

The basic course has been divided into three modules/ 
sections (major subject areas), the first contains 20 lessons, 
the second 15 lessons and the third two lessons. Practical 
@eplication is required in lessons 38 and 39 while a full 
course review is provided in the last lesson. Each lesson 
consists of a series of small blocks of information. A 
question is asked at the end of each block to check compre- 
hension and progress, providing immediate feedback to the 
student. Each question has several responses from which to 
choose and the student must select the answer believed to be 
best. The student is then either directed to a page from the 
lesson book or presented with a computer image which contains 
a comment on the selected answer stating whether it is cor- 
heet Or ancorrect, and why. After providing remedial infor- 
mation following an incorrect selection, the student is 
directed to choose another answer. If the answer is again 
incorrect, the cycle continues until the student selects the 
Gon’ eect amowem. Im the written format, the order of the ma- 


terial, questions and responses is scrambled so that it is 
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not possible to skim the pages of each lesson book and there- 
by skip the progress checks. Design of the computer version 
also eliminates "skimming". There are no "End of Module/ 
Section or Lesson Progress Tests". 

wach lesson is developed in a story line. The central 
character is a junior officer (Lieutenant) who is seemed 
@naeuntrained in human factors. In the beginning, the lieu- 
tenant is characterized as having a negative attitude towards 
human factors. As the course continues, various events cause 
the officer to become a proponent of Sumeia Epetors. It is 
intended that the student learn from the numerous mistakes 
of the ficticious Lt. I. M. Eager and the guidance provided 
by his HFE mentor and boss, Capt. B. Smart. 

In addition to the data presented throughout the lessons, 
a student supplement has been provided. As the student pro- 
gresses through each lesson, he/she is asked to refer to the 
Supplement. The supplement provides the student with graphs, 
photos and charts, some practical work (mathematics and 
drawings), and "nice-to-know" information. 

The course has been designed with the intent that each 
Should take no longer than one hour to complete, with an 
average completion time of 35 - 40 minutes and should be done 
on a daily basis. Therefore, maximum total completion time 
Should not exceed 40 hours, or 40 working days if one lesson 
is finished per day. It is anticipated that the computerized 


version would, in all probability, reduce the completion time 


25 





fer each lesson, as it eliminates the procedure of flipping 


through the lesson books searching for progress. check re- 


Sponses and new material. The-computer locates the response 


and displays it automatically for the student. 


In addition to the aforementioned course materials vari- 


ous MIL-STDs, handbooks and documents are provided to the 


Student and their usage is addressed throughout the course. 


Some are required for successful course completion®, while 


3 a: : 
Documents required for successful course completion: 


a. 


MIL-H-46885B. Human Engineering Requirements for 
vai ste Systems, Equipments ah Facilities. Janu- 
any ©2979. 


MIL-STD-1472C. Human Engineering Design Criteria 
for Military Systems, Equipments and Facilities. 


MIL-STD-1747B. Noise Limits for Army Materiel. 
Mareh 1975. 


MIL-HDBK-759. Human Factors Engineering Design for 
Army Materiel. Marc 

Army Regulation 602-1. Personnel-Materiel Systems: 
Human Factors Engineering Program. U. S. Army 
Headquarters, Washington, D. C., June 1976. 
Technical Memorandum 29-76. Guide for Obtaining and 
Analyzing Human Performance Data in a Materiel De- 
velopment Project. erson, Bb. L. an fours, W. H., 
U. S. Army Human Engineering Laboratory, Aberdeen 
Proving Ground, Md., September 1976. 
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others are referred to as additonal resources’. While thie 
Dasic package of reference documents provided with the course 
materials must be returned along with the lesson books to the 
issuing agency, they may be ordered through the standard mil- 
itary supply system. A brief summary of each of these re- 


sources is provided in Appendix A. 


Gee COURSE PRESENTATION 

The course materials are presented in a manner which 
allows the student to complete the course of instruction 
without the aid of an on-site instructor. The student sup- 


plement is intended to be retained by the student and used 


+ Documents referred to but not required for course com- 
pLetion : 
a. MIL-STD-721B. Definitions of Effectiveness Terms 


for Reliability, Maintainability, Human Factors, and 
safety. August l : 


b. FED-STD-595. Colors. 26 August 1964. 

c. Holshouser, E. L., Guide to Human Factors Engineer- 
ing General Purpose Test Planning (GPTP). Pacific 
Missile Test Center, Pt. Mugu, Ca. 

d. Human Factors Test and Evaluation Manual (HFTEMAN) , 


loco ie and 111, Pacitic Missile Test Center, 
Peewee, Ca. 


e. Human Factors engineering Data Guide for Evaluation 
(HEDGE). U.S. Army Test and Evaluation Command. 


20 December 1977. 


f. Mission Operability Assessment Technique: A System 
Evaluation Methodology (MOAT). October 1979. TP-79- 
31. Pacific Missile Test Center, Pt. Mugu, Ca. Lt. 
Vowemencumeana @, Lb. Donnell, Ph.D. 
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iow seand-alome” resounee and reference document following 
‘course completion. The student may contact the issuing agency 


in the event additonal help is required. 


D. COURSE LEARNING OBJECTIVES 

In accordance with the recommendations of NAVEDTRA 106A, 
a list of performance requirements was collected by course 
designers after consultation with human factors engineering 
test and evaluation personnel. These were then restated into 
task requirements (Appendix B) and forwarded to the civilian 
contractor who was assisting in the course development. The 
course learning objectives (Appendix C) were then developed 
from this list of task requirements. No module or lesson 


fearming Ojbectives were developed. 


Peo URSE CONTENT 

As described in the introduction of the Student Supple- 
ment, the course is divided into three major content sections 
(modules): (1) human capabilities and limitations, (2) human 
integration with the system, and (3) "Human Factors in the 
Military". 

Module I (Lessons 1-20) investigates the history and con- 
pmiltcaemecd 10r human factors engineering. Physiological ca- 
pabilities and limitations of human beings and their 
incorporation into the proper design and controls and displays 


are addressed. Also included is a discussion of the human's 
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interactions with the surrounding environment and the effects 
it may have upon his/her performance. 

Module II (Lessons 21-35) discusses: (1) the role of the 
human engineer in various aspects of analysis, 1.e., systems, 
cost, task analysis and etc., (2) selection and training of 
personnel; and (3) experimental techniques and statistical 
Goncepts. 

Module III (Lessons 36-37) focuses on human factors organ- 
izations, documents, and future applications. To. provide 
Students with a practical application of what has been pre- 
sented in the course, lessons 38 and 39 present a "real world" 
problem for solution. Finally, in lesson 40, a more typical 
review of the entire course is conducted. 

The outline of the Module and lesson topics is provided 


in Appendix C. 
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III. EVALUATION PLAN 


The validation process is a critical step in the devel- 
Opment Ot lesson materials. The question to be asked is: 
"How can we be certain that the instruction works?" The 
only acceptable way to do this is to measure student per- 
formance. If the students learn the specified skill, 
task, or knowledge as a result of the instruction, then 
Te is valid; if the students do not learn as specified in 
the learning objective, the instruction is not valid and 
must be revised. Validation can’'be compared to the test- 
mre process on a new piece of equipment. If particular 
components of the equipment malfunction repeatedly, then 
this indicates a problem that must be corrected if the 
equipment is to perform properly. By the same token, if. 
the students fail to master a particular segment of in- 
struction, a problem must be corrected. [Ref. 11] 


Memdetine the strategy used to assess the potential effec- 
tiveness of the instructional materials for the HFE course, 
an evaluation plan was developed (Appendix E), in accordance 
with the.guidelines provided in NAVEDTRA 110A. The evaluation 
plan provided in this thesis combined several of the valida- 
tion steps prescribed. These combinations were made because 
(1) no validation plan had been developed by the course spon- 
sors, (2) no previous evaluation of the course materials had 
been conducted, and (3) the course was in an advanced state 
of development. While the computer version has been written, 
it is not available on a commonly accessible system. The 
evaluation was, therefore, developed using the set of written 


manuals. 
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A. HOW TO CONDUCT VALIDATION 
Validation is accomplished by completing the following 
steps [Ref. 11]: 
1. Interval Review. As the rough draft materials are 
evaluated, the following questions should be asked: 
ase eiswthe content accurate? 
Db. ier the aaa inomemenned in a logical fashion? 
c¢. Does the lead-in information motivate the student 
Geepursue the material? 
d. Do the teaching-learning activities encourage 
productive learning? 
| e. Is the material written in a manner to allow 
maximum student participation? 
f. Is there an opportunity for review and practice? 
g. How effectively does the material teach behaviors 
specified in the learning aERCmEINeER 
h. Does the test measure the behavior as specified 
in the learning objectives? 
The internal review does not determine validity of the 
course material. It yeegn coueee” identify problem areas 


and suggest alternatives. 


fT elicivediemnrals. Informal, individual trials on 
rough draft materials for each lesson will take place to 
identify gross deficiencies in the lesson materials ''before" 


expensive final draft materials are produced. [In other 
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words, these trials Should help identify where more practice 
is needed, instruction is adequate, and ee instructor 
guides or student study booklets are complete. Since this is 
probably the first time students will actually take the test 
items developed for a lesson, the informal trial may also 
identify problem test items. 

The individual trials should be conducted as follows: 

a. Administer Pretests. Pretests are given to stu- 
dents to determine how much they know before being exposed 
to the instruction. The scores made.on the pretest are then 
compared math scores made on post tests to ascertain the ef- 
PMeeceinstructionai materials had on student performance. 

Tests developed for validation purposes are com- 
posed of the content items that address both enabling and 
terminal objectives. For validation purposes, however, it 
may be helpful to add test items to both pre and post tests 
which would be diagnostic in nature and therefore help to 
locate problem spots in the instructional materials. 

b.. Administer Instructional Materials. Observe the 
students' performances as they complete the lesson and record 
their completion time, the questions asked, and any difficul- 
ties observed. 

c. Administer Post Test Items. This will measure 
mastery of the content by the students. The results will be 
used to identify strengths and weaknesses in the course and 


to provide a basis for making revisions. 
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d. Administer Student Feedback Sheets. This is the 
student's opportunity tio suggest how to make instructional 
materials more effective. As such, the feedback sheet should 
be structured so that students’ responses are channeled, yet 
flexible so that open ended responses are allowed. 

e. Revise the Instructional Materials. Evaluate all 
data: test scores and instructor and student comments. Then 
make the revisions required. When all major deficiencies have 
been corrected and revalidated, the individual trials are 


concluded. 


e onoup Irials. A minimum of six students should be 
used for each group trial. As with individual trials, it is 
very important to select students that are representative of 
the target population and that have the proper prerequisites 
for the experimental group. The steps for conducting group’. 
trials are the same as were followed during the individual 
trials with one major exception: this time, there is only 
Gime Observer to six Or more Students. As students proceed 
through the instruction, they are not interrupted; they con- 
tinue through it as would a normal class. Only upon comple- 
tion of the lesson materials may students be asked such ques- 
tions as why they missed certain items, did the instructional 
materials maintain their interest, were there enough practice 


items. 
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4. Operational erosion: Validation of a new or revised 
eousse will be conducted in the operational environment using 
the normal student population. The reasons for conducting 
the operational validation are: | 

a. Instructional materials must be evaluated as an 
integral part of a total system. Until now, individual aad 
group trials validated portions of instructional material in 
an isolated environment. 

b. Analysis of data from this larger sample will 
provide a solid basis for final revision and refinement of 
the course. Data gathered at this point will provide feed- 
back concerning the adequacy of the learning analysis, learn- 
ing objectives, criterion tests, and instructional materials. 
If students fail to meet the Santa not the terminal objec- 
tives, reassessment of each of these procedures and products 
of the course Slectee and development processes shall be scru- 
cimnized. 

c. An operational trial provides an opportunity to 
work out administrative, equipment, facility, or any other 
implementation problems which may cause trouble later. 

The steps for completing operational validation are: 

(1) Review of Material by Functional Command. 
The functional command will review a cross-section of all 
instructional materials for content, technical accuracy, and 
format. These will include instructor guides, student study 


guides, remediation guides, media, student study booklets and 
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ie owicony guides. Discrepancies noted by the functional 
command will be discussed with the curriculum designer. 

(2) Administer Pretests, Instructional Materials, 
and Post Tests. As with both dear aanad and group trials, it 
1s absolutely necessary to have students with the appropriate 
emery level; without this, validation results are contamina- 
aed . 

(3) Administer Student and Instructor Feedback 
oneets. 

(4) Analyze Results and Display/Present Data to 
Supovel Authority. After administering all the tests, in- 
Struction, and feedback sheets, the results must be analyzed 
and displayed for submission to the approval authority. Pri- 
or to operational ae eons a validation standard must be 
set. At this time, the validation test results are compared 
with the validation standard. I£ the standard is not reached, 
Pueweurriculum designer should look carefully at the instruc- 
tional materials, equipments, and management documents to 
locate the problems and decide on corrective action to be 


taken. 


B. EVALUATION PLAN 

The format used for the Evaluation Plan was modeled after 
that used by the original test site in evaluating equipment 
and systems [Ref. 5] because of familiarity with selected 


test site supervisors. Following its design and development, 
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other military branch sites requested to be included in the 


evaluation. A summary of the evaluation plan follows: 


iP bugpose. «The purpose of the evaluation was to pro- 
vide an assessment of the potential effectiveness of the HFE 
Self-Paced Course and its potential value for use in the mil- 
ay test and evaluation community. The evaluation was plan- 
ned in cooperation with the HFE Laboratory at Pacific Missile 


Test Center, Pt. Mugu, California. 


weeeoaexcround. Ihe HFE Self-Paced Course was developed 
to satisfy a need for increased awareness and more indepth 
understanding of human factors. The Government Accounting 
Office (GAO) highlights this need and identified a defici- 
ency in the performance of various weapon systems 
...because the DOD does not pay enough attention to lo- 
gistic support, human factors and quality assurance during 
the design phase of the acquisition process. These pro- 7 
blems deter the systems' effectiveness to defend our coun- 
try in case of war. GAO, therefore, makes recommendations 
to improve the management and planning of ownership con- 
Siderations that have an impact on the effectiveness of 
a weapon system. [Ref. 1: pp. 1-27] 
The Human Factors Test and Evaluation Manual (HFTEMAN) 
Vols. I, II and III was distributed to various government 


agencies in October of 1976 by Pacific Missile Test Center. 


This document was developed to provide standardization in 
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procedures, testing and criteria in evaluating human factors. 
A SaGiic knowledge of human factors ‘terminology, principles 
and concepts was necessary for its most effective use. This 
was Pierally lacking and therefore HFTEMAN was not utilized 
as it was intended. 

| The HFE Self-Paced Course — developed to provide 
this basic knowledge. It has not been evaluated or used on 
a trial basis in any portion of the military prior to this 
Study. 

iis reseameher was tasked to (1) develop an evalu- 

ation plan in accordance with NAVEDTRAs 110A and 106A, (2) im- 
plement it, aia the test in a realistic environment, 
and (3) analyze the data and provide results, recommendations, 
and conclusions. Additional taskings and responsibilities 


are described in Appendix E. 


3. Prevalidation. This researcher conducted the inter- 
nal review with the assistance of content and curriculum 


Geveropment subject matter experts. 


4. Informal/Group Trials/Operational Trial. 


a. Personnel Selection. Subjects participating in 
this evaluation were drawn from four military commands (two 
Navy and two Air Force) whose primary mission was operational 
test and evaluation of equipment and systems prior to their 
acquisition by DOD for service use. Personnel were to parti- 


Cipate on a volunteer basis. Students selected from the 
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volunteers were to be representative of the Par ger population 
which om be ultamate ly usang the instructional materials. 
The following personnel prerequisites were established: 

(1) Subjects were selected from various warfare 
communities, 1.€., air, surface, subsurface and special war- 
fare . 

(2) No constraints regarding rate, rank, speci- 
ality code, grade, educational level or prior experience were 
imposed. 

(3) Each subject was required to be actively 
involved in planning and performing test and evaluation. 

b. Testing Constraints. 

(i) elbedunee Comter Instructor. No on-site 
instructors were available at any of the fest sites, nor 
were they required. This researcher was designated as the 
Students’ contact point, temporarily fulfilling the future 
role of the sponsoring agency. A Command Course Supervisor 
(CCS) was identified at each location to assist this re- 
Searcher in distributing and collecting course. materials, 
tests, feedback sheets (questionnaires), and recording stu- 


dent questions or comments and demographic data. 


(2) Personnel Selection. While HFE designers 
and evaluators are equally important, the availability of 
inexperienced equipment designers needing instruction was 
extremely limited as compared to those in equipment evaluation. 


The very dynamic and mobile nature of the course designers' 
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military target population identified the Imbendedy» users of 
the course--U. S. military operational test directors involved 
in whole or in part in human factors evaluation. The original 
test plan called for the use of volunteers only, however, due 
to limited personnel availability, final analysis showed that 


some participants had been assigned. 


(3) Course Materials. Original plans allowed 
60 days for student completion of all 40 lessons of the HFE 
Self-Paced Course. As a result of delayed procurement of 
course materials, the actual evaluation was limited to 30 


days and the first 20 lessons of the course. 


(4) Course Completion. The personnel partici- 


pating in this evaluation were also performing their normal 
jobs as test directors. Potential conflict was anticipated 
with their normal responsiblities which could require their 
absence from the command for a prolonged period of time and 


cause interruption in course progress or even termination. 


(S)- Learning Center. No area was available at 
any location which could be specifically designated for stu- 
dent use. Students studied in offices, libraries and at home. 
This was congruent with the course design and the intent of 


the study. 


(6) lteseme. Nowtestsevad been developed by the 


sponsor. The tests used in this evaluation were developed by 
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the author and had received only limited pretesting prior to 
the evaluation because of time constraints and therefore had 


not been validated. 


(7) Training. No training beyond that offered 
by the HFE Self-Paced Course was conducted during the opera- 


tional trial run of this study. 


oe Judgement Criteria. Both the pretest and post test 
were designed in accordance with the objectives developed by 


this researcher (Table 2). 


a. Pretest. The pretest was used to measure prior 
knowledge of the course content possessed by entering students. 
The pretest was composed of four different types of questions, 


” 


each being assigned a specific point value for scoring: 


multiple choice 1 point per answer choice 
‘matching 1 point each response 
fill in the blank. 2 points each 

Short answer 3 and 5 points each 


Tiesseores of the Course Group were compared to those of the 


Control Group to ascertain entry level equivalency. 


bB. Post Test. The post test was administered after 
completion of 20 lessons of instruction to assess whether: 
(1) the students had mastered the objectives of the first 
module, and/or (2) the students taking the course knew more 


about human factors than students not taking the course. The 
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Table 2 


Learning Objectives Developed (By the Researcher) 
For Design of Pre and Post Tests 


Given a list of human factors terms, write a definition 
for each in your own words. 


Given a list of specific HFE probems and/or areas of con- 
cerns, identify which human factors references should be 
consulted to provide solutions and guidelines. 


Given at least one control/display example, use the ap- 
propriate humamw factors reference to determine which 
features need to be redesigned. 


oivem a list of potential sources of technical informa- 
tion on human factors, identify the general purpose of 
een . 


Explain in your own words the importance of evaluating 
the human factors aspects of systems and equipments. 


fdentify the human factors principles which must be ap- 
plied during the evaluation of a workspace and control/ 
display panel. 


Given the four phases of the Weapons Acquisition Cycle, 
MwescrPoe the HFE*specialist role, the major HFE concerns 
and objectives of testing for each phase. 


Given the mission statement and the three functions which 
must be performed for a particular system, determine the 
human performance requirements. 


Given a list of 10 factors and forces which affect human 
performance; identify what aspect of human performance 
Sacm affects. 


Given a list of 10 factors and forces which affect human 
performance, identify what unit of measure must be used 

to assess the amount of protection required to diminish 

their adverse effects upon the human. operator. 


Explain in your own words, the importance of performing 
a task analysis. 


Given a choice of 3 possible task analyses, correctly 
identify which satisfies the definition of "task analysis". 
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post test was composed of the same type of questions and the 
associated point values as the pretest; however, the propor- 
tionment of the four different types of questions was dif- 
ferent so as to reduce "chance" selection of correct answers. 
The post test was designed to test the same learning objec- 
tives as the pretest. The post test scores of the Course 
Group were compared to those of the Control Group to ete 
mine whether or not learning had occurred as indicated by 

the Course Group's scoring significantly higher than the 


Control Group. 


oe fest eens. Eaeh testeitem was "referenced" to 
a specific re uminial objective. Since each objective was a 
Statement of a "criterion" for student performance, these 
Meets 1Or Objectives were identified as "criterion-refer- 
enced” tests. The test items were developed in accordance 
With the guidelines provided in NAVEDTRA 110A. The questions 
and answers were extracted from the Human Factors Engineering 
Self-Paced Course, lesson 20 which contained review material 
covering lessons 1-10. Student responses which expressed 
the same answers as the course materials were given credit. 
Partial credit was awarded where possible. Test key answers 
were compared to the answers provided by selected HFE experts 
to ensure the HFE course answers were, in fact, correct. 

Additonally, to help identify weaknesses, test 


items were added to both pre and post tests which were 
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diagnostic in nature and therefore would help locate problem 
areas in the instructional materials. Results were recorded 


for each test item. 


d. Hypothesis for Comparison. If there was no 
Statistically significant difference between the post test 
scores of the group completing the first module and the scores 
of the Control Group which did not take the course, it would 
indicate that the course materials were not effective and 
would be of no value for use in the military. 

Further analysis would then be conducted on both 


the pretest and post test questions. 


e. Statistical Analysis. The statistical analysis 
was performed on the pre and post tests and was intended to 
establish (1) the entry level equivalency of both the Course 
Group and the Control Group based on the pretest scores and 
(2) to establish a difference in the post test scores between 
the two groups. The results of the post test scores would 
therefore provide an overall statement of the potential 


effectiveness of the HFE Self-Paced Course. 


f. Raw data for the pre-post test analysis for the 
operational tryout was provided to this author in conjunction 
with students' tests and is presented in summary form (Appen- 


dix G). 
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6. Standards 

Criterion to establish equivalency between both the 
Course Group and the Control Group on the pretest scores had 
been set by this author. There must be no statistically sig- 
nificant difference at the .025 level (two-tail t test) to 
determine that the groups are equivalent in their pre-entry 
Teme 1 . | 

Criterion to establish that learning has taken place 
with the Course Group had been set by this author. There 
must be a statistically significant difference between the 
Course Group and the Control Group post test scores, at the 
mum comzidence level (two-tadl t test) to determine that 
E@emeroups are not equivalent in their level of learning. 
To validate that the course materials were able to success- 
fully provide the stimuli for knowledge and skills transfer 
to the students taking the course, their post test scores 
must be significantly greater than the Control Group. 

Criterion guidelines were established by CNTECHTRA 
for acceptability of course objectives, design, presentation, 
availability, maintainability and supportability in NAVEDTRAs 
110A and 106A and are summarized by this researcher in Table 
3 and described in greater depth in Appendix E. 

In the event the standards are not met following the 
Operational tryout, instructional materials would be analyzed 


as indicated above. 
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7. Remediation 
Students involved in .the operational tryout receive 
remediation from the responses to the Progress Checks follow- 


ing each block of instruction within the lesson topics. 


S. instructor Training 
The HFE Self-Paced Course was intended for use by 


individuals without the aid of an on-site instructor. The 
intent of the agency sponsoring the course is to provide a 
point of contact for students to answer any questions on con- 
meme, procedures, etc., once the course is approved. There 
was, however, no instructor guide available during this study. 
If none is to be developed, the intended point of contact 

muse then be a subject matter expert. The student and the 
expert would then talk through any problems using the set of 


student course materials. 


Te cucent sand Instructor Feedback Sheets (Questionnaires) 


The following student, Command Course Supervisor and 
Curriculum Development Expert. feedback questionnaires were 
completed and returned to this author for review and analysis. 

a. Annex (A) of Appendix E was completed by each 
participant in the evaluation during orientation. 

b. Annex (C) of Appendix E was completed by each 
participant prior to the students beginning the course. 

c. Annex (D) of Appendix E was completed by each 
participant after the Course Group had completed the first 


20 lessons. 
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d. Annex (B) of Appendix E was maintained by each 
Command Course Supervisor (CCS) acu nontethe evaluation 
period. 

e. Annex (G) of Appendix E was completed by experts 
in the field of curriculum development. 

f. Annex (F) of Appendix E was completed by each 
Course Group participant upon completion of the post test. 

g. Annex (E) of Appendix E was completed by each CCS 


-upon completion of the evaluation period. 


10. Display Format 


Test analysis results are displayed as Appendixes F 
and G and discussed in Chapter IV of this thesis. 
Analysis of the questionnaires are displayed in 


Appendix H and discussed in Chapter IV of this thesis. 
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IV. METHODOLOGY AND RESULTS 


For this study 52 participants were chosen from four 
test sites, each of whom met the prerequisites for course 
entry as described below. These 52 participants were divided 
into two groups; one which would complete the course (Course 
Group) and the other (Control Group) which would not go 
through the course, but would be administered the pre and post 
tests. Since the pre and post tests were to be structured 
differently, the post test scores could not be compared to 
the pretest scores to measure the effectiveness of the course 
in providing the stimuli for transferring knowledge and skills 
in human factors. It was, therefore, necessary to eae 
Gaempose test results of the Course Group to those of a 
Control Group who had not completed the course of study. 
For that comparison to be valid, it was necessary to estab- 
lish an equivalency of the entry levels of the two groups 
prior to’ commencing the course of instruction. This measure 
of equivalency included data gathered from demographic infor- 
mation sheets completed by each participant and pretests on 
the subject material. 

Of the original 26 students beginning the course, 10 were 
unable to complete the 20 lessons due to conflicts of tasks 
imposed by their job responsibilities. Therefore, only 16 


post tests from the original 26 students in the Course Group 
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were used for the oe All 26 post tests from the Con- 
trol Eneup were used. A breakdown of experiment participants 
and data sources is provided in Table 4. 

The comprehensive examinations used for the pretest and 
post test were prepared by this researcher (Appendix E, An- 
nexes C and D) and structured as described in section III Be 
this thesis. While the post test had the same type of ques- 
tions as the pretest, there were a larger number of short 
answer and essay type questions on the post test. This tend- 
ed to eliminate a large portion of guessing and required 
that the students have a more comprehensive knowledge of the 
course material. [Ref. 11: pp. 3-30] Although no claim is 
made as to the validity of these examinations, they do 


represent a measure of student knowledge and achievement in 


relation to stated learning objectives of the course. 
A. DEMOGRAPHIC DATA, ANALYSIS AND RESULTS 


No prerequisites were required of students prior to 
taking the HFE Self-Paced course. A prerequisite established 
by this researcher for this study required that each student 
be actively involved in test and evaluation for motivational 
purposes. 

A Student Information (SIF) sheet was distributed to and 
completed by each participant. Specific information included 
pay grade, educational level, time stationed at the test sites, 


job description and previous courses completed in human factors 
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traming. this demographic data for each participant is pro- 
vided in Appendix F. A comparative breakdown (Table 5) be- 
tween both groups was performed to specifically identify 
demographic ee eae which could bias the results of 
this study. 

See distribution of mised personnel and eee 
between both groups was disproportionate with the Course 
Group having 50 per cent enlisted and 38.5 per cent officers 
compared to the Control Group's 23.1 per cent enlisted and 
65.4 per cent officers. The educational level of the two 
groups was also unbalanced with 80.8 per cent of the Control 
Group having college and/or Peeecacnade degrees compared 
to the 57.7 per cent of the Course Group. Specifically sig- 
Micant are the percentages of those having only high 
school diplomas; 3,8 per cent of the Control Group, compared 
to 34.6 ee eomt Of dhe Cannes Gromp, Addatronally, the 
Control Group had an average of 35.9 months experience in 
test and evaluation while the Course Group averaged only 17.2 
months. There was also a disparity in the number having 
participated in the test director's course offered by two of 
the test sites; 76.9 per cent for the Control Group and 53.8 
hen cGCeme fOr the Course Group. All participants in this 
experiment were actively involved in test and evaluation. 

These results caused some concern that test scores would 


be biased since the Control Group as a whole seemed more 
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experienced in performing test and evaluation, educated to 
a higher level, and-more experienced in middle eaten 
responsibilities for test and evaluation. The actual re- 
sults of the pre and post tests, however, did not substanti- 
ate this concern. 

Additionally, one demographic percentages between the 
Gmaginal 26 and the final 16 members of the Course Group who 


completed all 20 lessons vary insignificantly. 


B. PRE-POST TEST ANALYSIS AND RESULTS 

oem ali test participants had completed the pretest, the 
eombbined scores of all participants (Appendix F) in each of 
the two groups were compared. The mean score for the Control 
Group was 70.19 with a standard deviation of 14.85. The 
mean score for the Course Group was 64.73 with a standard 
deviation of 18.73. The following pooled variance formula 
for the t-test was used to test the Significance of “the dif- 


ference in group mean: 





Popham and Sirotnik state that this formula will result ina 


t-value which may be interpreted with more degrees of freedom 
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than other t-formulas. [Ref. 12: p. 139] Therefore, since a 
smaller t-value is needed to Rect a given null hypothesis 
when a greater number of degrees of freedom are used, this 
#0rmula produces a t-test that is more likely to be signifi- 
cant and, as a result, it is a more powerful test. 
| Hypothesis one assumes that the students composing the 

two groups do have an equivalent knowledge level prior to the 
entry of the experiment and, therefore, there would be no 
Significant differences on the pretest totals of the two 
groups. The results of the analysis of scores are presented 
ma heble 6. 

The computed t-value was compared with a distribution 
Of t-tables from the Students t-table. The values found 
indicated that for 50 degrees of freedom, the t-value of 1.74 
at the .05 level of significance: should be used. ([Ref. 13] 

The computed value of t for null hypothesis one has led 
to its rejection and acceptance of the alternate. Therefore, 
there was no significant difference between the pretest scores 
of the Control Group when compared with those of the Course 
Smeup. - [herefore, comparing the performance of the two 
groups on the post test to detect any improvement by the Course 
Group is acceptable. 

Hypothesis two states that there would be no equivalency 
between the post test scores of the Control Group and those 


of the Course Group, with the scores of the Course Group being 
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greater than those of the Control Group by at least one 


‘standard deviation. This hypothesis assumes equivalency in 


the scores on the pretest by both groups. 

When all participants had completed the post tests, the 
combined scores for both goups were compared. The mean post 
test score for the Control Group members was 49.70 with a 
standard deviation of 16.48. The mean post test score for 
the Course Group students was 93.39 with a standard devia- 
tion of 25.14. This analysis (Table 7) resulted in a sig- 
Meticantly high t-value. The difference between the Control 
Group and Course Group was demonstrated and therefore the 
null hypothesis two was rejected. 

The post test results derived from cummulative scores of 
all participants (Appendix G) are provided for further study 
and test development. A total of 176 points was possible on 
Piempost test Which covered content in the areas of (1) task 
analysis, (2) selection of personnel for testing, (3) work- 
Space design and arrangement, (4) control panel design, 

(S) individual information processing capabilities, (6) envi- 
ronmental factors which may affect human performance - units 
of measure, (7) environmental considerations inclusive in 
design, (8) objectives of taking a human factors course, and 
(9) factors contributing to human error. 

A summary of these results for the Course Group (Table 8) 
Shows that all content area scores were at least 50 per cent 


of the total possible points with the exception of "Control 
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Panel Design", "Individual Processing Capabilities", and "En- 
vironmental Factors Affecting Human Performance" which 
were rated as 46.38 and 48 per cents respectively. 

A summary of these results for the Control Group (Table 8) 
shows that only one content area (Factors Which Cause Human 
Error) scored at least 50 per cent. All others scored less 
than 50 per cent with a range of 3-48 per cent. 

Additionally, the test questions were redistributed under 
each of the enabling objectives (Table 2) which they support- 
ed: 

1. Given a list of human factors terms, write a defini- 
tion for each in your own words. 

2. Given a list of specific HFE problems and/or areas 
of concerns, identify which human factors references should 
be consulted to provide solutions and guidelines. 

3. Given at least one control/display example, use the 
appropriate human factors reference to Jenene which features 
need to be redesigned. 

29. Explain in your own words the importance of evaluat- 
ing the human factors aspects of svstems and equipments. 

6. Identify the human factors principles which must be 
applied during the evaluation of a workspace and control/ 
display panel. 

9, Given a list of 10 factors and forces which affect 
human performance, identify what aspect of human performance 


each affects. 
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10. Given a list of 10 factors and forces which affect 
human performance, identify what unit of measure must be used 
to assess the amount of protection required to diminish their 
adverse effects upon the human operator. 

All the scores for the Course Group (Table 9) rated at 
ease 50 per cent with the exception of Objective 6 which 
rated 45 per cent and Objective 9 which rated 49 per cent. 

The Control Group scores (Table 9) failed to reach 50 
per cent with the exception of Objective 6 which rated 5l 
memecent. 

Since the test questions had received only minimal pre- 
testing, the results warrant further test item development 
and evaluation. Although no specific conclusions will be 
reported based on this preliminary analysis, it can be 
noted that the Course Group obtained higher scores than did 
the Control Group in all but one of the content areas and in 
all of the enabling objectives. (See right columns of Tables 


8 and 9). 


C. QUESTIONNAIRE ANALYSIS 

Three separate questionnaires were designed and adminis- 
tered in accordance with NAVEDTRAs 106A, 110A and the Ques- 
tionnaire Construction Manual developed for the Ft. Hood Field 
Unit by the U. S. Army Research Institute for the Behavioral 
Social Science. Summary information on the design and use 


of each questionnaire is provided in Table 10. Every effort 
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was made to include questions on the rating forms that the 
specific users wens best qualified to answer. 
1. Student Attitude Survey (Questionnaires) 

Students who had been able to complete at least five 
lessons were required to complete the Student Questionnaire 
On various aspects of the self-paced instruction received. 
There were 24 students in this classification. Students were 
directed to answer only those questions that were applicable 
to the material they completed. Therefore, the reader will 
note that the total number of responses for each question may 
tend to vary. 

Rating scales employed in the student questionnaires 
were based on a balanced five-point Likert Scale. The nun- 
ber of students selecting each descriptor was tabulated. 
Ordinal values of l Ors were given to student ratings, with 
5 being the most positive and 1 being most negative. These 
ratings were then divided into three groups and interpreted 
as "positive (5-4), "neutral" (3), and "negative" (2-1). 

The frequency of students responding to the two most positive 
responses was combined as was the frequency of responses for 
the two most negative. These figures were then translated 
into a percentage of the total number of students responding. 
The mean and standard deviation of the students' responses 
for each question were also calculated using the formulas 


listed below: 
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The mean represented the degree of positiveness of the Course 
Gromp'S responses to each question. These responses were 
used to measure their attitudes and assist in the assessment 
of the various aspects of the course and administrative pro- 
cedures such as: 

a. Course Content and Objectives 

b. Course Design 

€. VGourse Presentatzon 

a. jJStudentwSoupphement 

e. Resource Documents 

f. Human Factors Attitudes 


g. Miscellaneous 


2. Command Course Supervisor (CCS) Questionnaire and Log 


Once CCS was selected from each of the four partici- 
pating commands. The CCSs maintained a log throughout the 
evaluation period and upon its completion filled out a 


questionnaire. 





Rating scales employed in these questionnaires were 
based on a numerical cell interval frequency scale. The fre- 


quency of responses was grouped into three cells: 


0 - 40% 
41 - 60% 
or - FOU% 


The CCSs estimated the frequency of student questions and 
comments that dealt with specific areas for each of several 
pertinent sections. The purpose of the data collected was 
to verify and clarify student questions, comments and sub- 
Stantiate the use of the course materials without the aid of 
an on-site instructor. CCSs were also directed to assess 
Student attitudes and behaviors during the evaluation 
based upon their observations, student interactions and log 
entries. These responses were used to assist in the assess- 
ment of the following areas: 

ae Lesson Book Materials 

Be oeuden t Supplement 

Seenccoumec Documents 

d. Motivation and Attitudes 

e. Student Information 


f. Additional Comments 


5. Currvicu lem Development Exwert Questionnaires 


Three proven specialists in the field of curriculum 


evaluation who regularly apply the standards and criteria 


es 





provided by NAVEDTRAs 106A and 110A were asked to assess the 
extent that each of the terminal objectives satisfied the 
Six criteria identified in Appendix E. The average number 
of years of experience for these experts was 5.5 years. All 
were Civilians in government service whose full time job was 
to evaluate various aspects of curriculum prior to its vali- 
dation and approval for use in the instructional systems of 
Navy schools. 

Rating scales employed in this questionnaire were 
based on a balanced four-point Likert Scale. No neutral 
response was provided.’ This scale measured the degree to 
which each terminal objective met each of the six criteria 
listed in Appendix E. Ordinal values ranging from 1 through 


4, were applied to the verbal descriptors as follows: 


4 3 2 sil 

/ vf / / 
Completely Mostly To Some Not At 
Degree All’ 


D. INTERNAL REVIEW AND QUESTIONNAIRE RESULTS 

The results for the Internal Review summarized in 
Chapter IV and specifically discussed in Appendix E are based 
On questionnaire responses from students and curriculum de- 
velopment experts as Welteas tiis researcner's personal eval- 
uation. 

Hypothesis three assumes that the Course Terminal Objec- 
tives, Design, Content, and Presentation would not fail to 
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meet the standards and criteria in accordance with NAVEDTRAs 
110A and 106A. 

Hypothesis four assumes that the course materials would 
not adversely affect student attitude regarding the need for 
human factors engineering test evaluation. 

As is evidenced in the results which follow, the alter- 
nate hypothesis three for Course Design is rejected with the 
failure of the Learning Objectives to follow a logical pro- 
gressive sequence which is supportive of the order of the 
content and lesson topics in Module 1. The layout and format 
are deficient in the following areas: (1) information prior 
to the regular progress checks at the end of the "information 
blocks", (2) remediation for selection of "wrong answers", 
(3) professionalism in the quality of the story line, (4) in- 
formation in the student supplement to allow its independent 
usage as an HFE test and evaluation planning guide, and | 
(5) ease and usability of the course lesson books caused by the 
"nage flipping" aspect of the course procedures. 

The null hypothesis three for Course Content, Presentation, 
and Design (sequencing of knowledge and skills factors con- 
tained in Module one) is rejected in as much as these areas 
Satisfied the criteria stated in Appendix E. However, while 


the lesson topics in Modules 2 and 3 are not officially part 


a ; 
of this review, it must be noted that the content failed to 


provide prerequisite knowledge and skills required for under- 


standing and demonstrating follow-on concepts and skills. 
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The alternate hypothesis four was rejected in a much as 
92 per cent of the students upon course completion, responded 
Eiae FE test and evaluation was “extremely important" (Ap - 
pendix H). The students' attitudes toward human factors 


testing were not adversely affected. 


im Course Learning Objectives (LOs) 
a. Job Task Analysis (JTA) 


No approved JTA was used to develop or evaluate 
the LOs, as none was available. The sponsoring military 
commands developed a list of task requirements (Appendix B) 
obtained from interaction with human factors engineers. This 
list was provided to the course designers to guide the 
development of the course. This procedure is acceptable 
when no JTA exists. 

Students assess the importance of the LOs by 
identifying those which they considered necessary in the 
planning and designing of HFE test and evaluation. The 
results are sunaeed in Table 11. With the exception of 
LOs 3, 4, 8, 10, 14, and 17 all the remaining LOs were judged 
by at least 40 per cent of the students to be important in 
planning and designing for HFE test and evaluation. 

Students also assess the importance of the LOs 


by identifying those which they considered necessary in the 
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performance of er job, which for most was general test 
and evaluation. [Ref. 3: pp. 36-38, 79-80] The results 
are summarized in Table 12. With the exception of LOs 8, 10, 
Dietz, and 17 at least 40 per cent of the students judged 
the remaining LOs as important. 
Db. “LOeGrateria 

The Curriculum Development Experts (CDEs) were 
given a questionnaire which provided a list of course LOs and 
asked to rate the extent to which each LO met the six previ- 
ously specified criteria (Table 13). According to the results 
of their evaluation, no objective met the criteria "Mostly" 
eee compietely'’'. It would appear that the LOs did not | 
(1) state the objective in terms of the learner, (2) state 
standards, and (3) state the conditions under which learning 
Peeeomoectir, Most of the LOs were not behaviorally specific. 
Raw data is provided in Appendix I. | 

The alternate hypothesis three is therefore re- 
jected with the failure of the course terminal objectives 
to meet the criteria identified in NAVEDTRA 106A and the 
absense of module (unit/section) and lesson enabling objec- 


tives. 


Z. Course Design 


a. Sequence and Structure 
Of primary importance in support of the "building 
block" concept of course development is the logical progres- 


Sion of knowledge and skill factors addressed by the terminal 
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objectives and the course content. This also assumes that 
the terminal objectives. will be presented in the same pro- 
gressive order as the lesson topics. | 

To conduct this evaluation, the intent of purpose 
of each learning objective was identified and broken down by 
this researcher into four areas: (1) knowledge of concepts, 
(2) knowledge of skills, (3) application of knowledge and 
(4) application of skills. The HFE course materials were 
then reviewed, identifying lessons which satisfied the in- 
tent or purpose of each objective (Table 14). NAVEDTRAs 110A 
and 110A require that a student be tasked to perform a skill 
only after the prerequisite knowledge factors have been pre- 
sented. This breakdown, therefore, enabled this researcher 
to evaluate and compare the logical progression of the termi- 
nal objectives and the lesson topics. Additionally, student 
attitude survey responses were assessed. 

In this researcher's opinion, all 20 lesson 
EGpacs within the first module appear to be supportive of 
the "building block" concept for development of knowledge 
and skills. Fifty eight and 66 per cent of student responses 
(Table 15) to questions Bl and B2 respectively are supportive 
of this opinion. In completing lessons beyond the first 
module, however, some discontinuity appears, i.e., statisti- 
cal concepts (lessons 33 and 34) such as probability of 
success, confidence levels and probability of failure-free 


system operation are addressed in lessons 22 and 26 
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respectively. Additionally, technical references (lessons 
36 and ay Such as HFTEMAN, HEDGE and MOAT are introduced 
in lessons 23 and 24. Students are told that more detailed 
information regarding these documents will be presented 
later. This researcher needed more information about these 
documents to clarify points being made in lessons 23 and 24 
and was required to search for the information in lessons 36 
and 37. It, therefore, becomes apparent to this researcher 
that modules two and three need to be reviewed to ensure 
that a logical progression of the knowledge and skills 
Poecors 1S provided. 

In this researcher's opinion, the terminal ob- 
jectives do not follow a logical progression as exhibited by 
Peewee +, specifically, LO 3 which required students’ 
knowledge of HFE technical documents as presented in lessons 
36 ad 37, 1S sequenced before LO 4 which nesses HFE goals 
wemmmesented in lesson 3. Therefore, LO 3 1s out of sequence 
or the material in lessons 36 and 37 should be presented 
earlier in the sequence (prior to lessons 23 and 24). Table 
ieereseics Other Examples of incongruencies of lesson 
materials. Additionally, at least 50 per cent of the students 
responding to question Al (Table 16) felt that objectives l, 
2, 4, and 7 were addressed within the first 20 lessons. 

pim@e@emilvemrinrst 20 lessons are judged to be logi- 
cally progressive and the LOs are not, they therefore do not 


coincide in a supportive relationship. As additionally 
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required by NAVEDTRA 106A, each objective must be taught with- 
in the course of instruction. LO 11 required the student to 
"Calculate human performance reliability" but the course 
does not teach the student how to do so and it was not one 
of the pre-course requirements. 
b. Figures and Tables 
All figures and tables used in the Student Sup- 
plement were rated by at least 65 per cent of the student 
Course Group as "positive" (Table 17). Written comments 
were received from participants which stated that while 
figures and tables were good, without the lesson books to 
provide additonal explanation, many of them could not be 
fully understood. 
c. Symbols and Legends 
All symbology and legends used in the Student 
Supplement were rated by at least 63 per cent of the Course 
Group as "effective" (Table 18). No additonal comments were 
received from students. 
d. Layout and Format 
Results of student attitudes RRS eme 
effectiveness of the layout (Student Supplement, eight lesson 
books, and resource documents) are displayed in Table 19. 
Comments indicated that it did not hinder their learning 
experience and many indicated they preferred the way the 
lesson topics were divided into separate lesson books. They 


remarked that this added to the portability of the course 
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materials. Additonally, the students felt that much of the 
material in the Student Supplement had to be clarified by 
information in the lesson books. It would therefore, be 

very difficult to use the Student Supplement as a stand-alone 
reference and planning guide for HFE test and evaluation. 
They considered its value as negligable once the student 
lesson books were returned to the sponsoring agency upon 
Completion of the course. 

Regarding the format, no less than 50 per cent 
responded positively to (1) Course Procedures, (2) Informa- 
tion Blocks, and (3) Quality of Questions. It must be noted 
that both course procedures and quality of story line re- 
ceived a negative rating by 42 per cent of the students. 
Written and verbal comments from participants indicated a 
strong dissatisfaction with "page flipping"' aspect of the 
course procedures to search for answers as well as an equally 
strong dislike for the story line following the antics of the 
meeticrous Lt. I. M. Eager. 

Addtional- amplifying information regarding layout 
and format were extracted from written comments on the ques- 
tionnaires and verbal comments recorded by the CCSs. These 
have been paraphrased or quoted directly when found to be 
representative of the majority of the course participants. 

(1) As a group, course participants felt that 
the information frames sometimes did not provide all the in- 


formation necessary to answer correctly the questions at the 


88 





end of the information frame. In fact, comments reported that 
sometimes the information was not given until the following 
frame. 

(2) The reinforcement provided in the "wrong 
answer frames'' was judged to be borderline. Students also 
commented that the wrong answer replies were demeaning, i.e., 
"Surely, you didn't answer that way". "A simple ‘correct' 
or ‘incorrect’ with some amplifying information will suffice." 
was a comment which was representative of a majority of the 
Students. | 

(3) The quality of the questions asked at the 
end of the “information frames" was considered borderline. 
Comments from students indicated that sometimes the questions 
could be answered by elimination of "ridiculous answer choices" 
while others had to be guessed.- Some participants felt 
that the questions were unclear or worded poorly so as to 
cause confusion. 

(4) Several students responded that problems 
such as those already mentioned as well as errors in the 
e@meceMateriais caused endless loops resulting in irritation 
and loss of motivation for continuing with the course. As 
Suemotudent Stated "I would have quit by lesson five, if I 
hadn't been assigned to complete the course." 

(5) While student questionnaires indicated the 
format was neutral, fifteen of the 24 students polled chose 


to write comments concerning their dissatisfaction with the 
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"page flipping'' format. They reported losing their places 
and having to begin again from frame.one of the chapter being 
Studied. This format may have prevented looking ahead, but 
meemest certainly "prevented review of any materials". Judg- 
ing from the forcefullness and emotion emanating from actual 
comments, the following quote sums it up, "In all the service 
schools and courses which I have Pee Hick and taken in the 
past years, programmed texts were an integral part of the 
curriculum. I have never seen a programmed text structured 
such as this one. [In all honesty, after the first 25 pages, 
my interest level dropped to zero and remained there. Thumb- 
ing through numerous pases tO find COrrect answers or incor- 
rect answers and then constantly repeating the procedure... 
created a frustrating situation that stifled any learning. 
This page shuffling drill detracted from any tangible bene- 
fits that might have been derived from the material presented.” 

(6) The inclusion of: a story line following the 
aagventures of the ficticious Lt. I. M. Eager and his buddy- 
boss, Capt. B. Smart was not accepted well by course parti- 
Cipants. Of all the questions answered, the vet feaiminzenes s 
of the story line was rated the lowest. While the numerical 
value of the mean places it at the "neutral" point, it does 
so by only .04 of a point away from "negative". 

(7) Fourteen of the students reported that the 

story sequence was sometimes disjointed and very much dis- 


tracting from learning the essence of the lessons. Overall, 
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they indicated a desire to "drop the junk and put more facts" 
"and real life examples in its place." "The stories made 

the text too 'cute', don't try to entertain me ina course, 
try to teach me. The way the material was presented (using 
Eager) just irritated me and the effectiveness of the course 


was diminished," 


oe Course Content 

In addition to the overall effectiveness assessment 
provided from the diagnostic test results, an important indi- 
Gator of the perceived value of the course content was the 
number of students desiring to keep copies of the Student 
Supplement and the lesson books (Table 20). Forty-two per 
cent of the course participants desired to keep a copy of 
the course materials and 54 per cent wanted to keep the 


meuaent Supplement. 


Table 20 


Student Attitudes Regarding Retaining a 
Copy of the HFE Course Materials for Future Use 





Percentage of Students 


Responses oer gine ae 

Yes No 

Want to keep course materials? 42% 58% 
Want to keep supplement only? 34 46 
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4. Course Presentation 

The percentage of student attitudes regarding the 
effectiveness of the course length (Table 21) was calculated 
with 52 per cent responding positively and 26 per cent neg- 
atively. Percentages of those regarding terminology used 
were 65 per cent positive and 9 per cent negative. Responses 
regarding participation in the course without the aid of an 
on-site instructor were relatively evenly distributed with 
37 per cent positive and 34 per cent negative. 

Regarding the latter set of responses, it is believed 
that contacting the CCS (who was not HFE trained) for ex- 
changing books and receiving tests and questionnaires may 
have been interpreted by students as contact with an on-site 
instructor. The CCSs' logs indicated that students seldomly, 
if ever, required their assistance with course content in 
Biemrole Of an on-site instructor. Additionally, this re- 
searcher received no communication from students asking for 
clarity of course instructions or content, even though it 
was encouraged in students' instructions.-. 

It must also be pointed out that 42 per cent of the 
course participants desired to keep a copy of the course les- 
son materials, even though they had been informed that no 
assistance would be available upon completion of the evaluation 
Weriod. 

Additional amplifying information regarding course 


presentation was extracted from written comments on the 
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questionnaires so verbal comments recorded by the CCSs. 
These have been paraphrased or quoted directly when repre- 
sentative of the majority of the course participants: 

a. The level of detail presented Sear e ee the 
course materials presented no difficulty Pe ecoOmeent compre - 
hension according to student reponses in the pte reece 
Some comments were made which indicated that students would 
like to see an even greater detail for some areas, however, 
they were not specific as to which areas. 

b. No negative comments were received regarding the 
course length. The time required for course participation 
was not overburdening TOmencHin However, Students felt that 
the format caused time to be wasted which could have been 
spent on the course content. 

c. The only feedback students received while taking 
the Meee eourse was provided in the answers at the end of each 
information frame. No periodic self-check tests were avail- 
able with the course materials. According to the course 
sponsors, none have as yet been developed. The sponsors had 
intended that measurement of the students’ attainment of the 
course LOs would not be necessary as their mastery om ene 
content would become apparent as it was applied in the normal 
Merrtormance of their jobs in designing and implementing tests 
for evaluation of the human factors aspects of equipments and 


Sy >S5tems. 
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5S. Course Availability 


According to the course sponsors, original plans 
called for the U. S. Army to supply all necessary materials 
to its own personnel, while the cognizant Navy commands 
would fulfill Navy requests, once the course was finalized. 
However, present constraints on Biase and personnel to make 
corrections and implement the computer program may hinder, 


if not prevent, this effort. 


6. Course Maintainability 


Future plans of perceived needs in this area cannot 
be reported, as none were received from the sponsoring com- 
mands. However, even during this preliminary evaluation, its 
importance was-clearly noted. At present, course materials 
refer to specific pages of MIL-STDs and handbooks which 
caused some confusion to course participants when using MIL- 
omlei472C. The HFE course ies references to specific 
pages within MIL-STD 1472B. In the latest version of the MIL- 
SiD, the pages had been renumbered and students stated that 
there was some difficulty in finding the correct figure or 
information they were assigned to read. Additionally, the 
United States has converted their measuring system to metrics 
and this is not reflected within the course materials. 

7. Course Supportability 

Future plans of perceived needs in this area cannot 

be reported as none were provided from the sponsoring commands. 


At present, supporting commands remain to be identified. 


95 





V. CONCLUSIONS AND RECOMMENDATIONS 


A. CONCLUSIONS 

The Human Factors Engineering Self-Paced Course has the 
potential to be most effective, based on its demonstrated 
ability to successfully provide the stimuli for the transfer 
of knowledge and skills to the Course Group in support of 
the intent of the course originators' task requirements and 
terminal objectives. The incorporation of the recommendations 
provided below would provide the course the potential to pass 
a validation attempt in accordance with CNTECHTRA requirements. 

Further validation of course content validity should in- 
clude a study of the Course Group participants after having 
had the opportunity to apply their newly gained knowledge 
and skills in the performance of their jobs in designing and 


conducting human factors tests. 


B. RECOMMENDATIONS 

It is recommended that the course IES, ar ois) jean 
comply with the standards and criteria identified by CNTECHTRA 
in NAVEDTRAs 110A and 106A for the development of military 
educational courses and incorporate the following changes into 


the Human Factors Engineering Self-Paced Course. 
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1. Course Learning Objectives 


Rewrite and rearrange course terminal objectives and 
develop lesson and module enabling objectives. Incorporate 
the elements required for acceptable learning objectives which 
appear to be missing from those presently given for this 
course: (1) specific Been cion Oo; expected learner .action, 
(2) conditions under which the action will take place, and 
(3) standards or criterion which must be reached by the stu- 
dents. Evaluate them again according to the six criteria re- 
quired by NAVEDTRA 106A. The 17 course terminal learning 
objectives have been restructured (Table 22) to meet the cri- 
teria. The learning objectives should, however, be presented 
in the same order as the course content. A recommended re- 
ordering of the course's original terminal objectives have 
been provided in Table 23. 

a Counse Design 

a. Sequence of Course Content. 

Reevaluate the sequence of the content as it is 
presently given to ensure that knowledge and concepts are 
taught before the students are required to apply them, l.e., 
task analysis before human performance evaluation. This will 
support the ‘building block" concept of learning skills and 
provide the student a better opportunity to attain success 
and avoid unnecessary frustrations. 

b. Layout of Student Supplement and Lesson Books. 

Follow the guidelines provided in NAVEDTRA 110A 
which addresses the proper physical layout and required 
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Table 22 . 


Examples of Recommended Restructured Terminal 
Learning Objectives IAW NAVEDTRA 106A 


Define in writing common terms used in human factors 
engineering in accordance with lessons 1-40. 


Given a list of HFE reference documents, select the major 
topics addressed by each in accordance with lessons 1-40. 


Given an HFE problem, demonstrate the ability to resolve 
at by correctly selecting the HFE reference and locating 
within its contents the section which provides the ne- 
cessary information in accordance with lessons 1-40. 


Describe in writing the basic goals of HFE in a materiel 
development program in accordance with lessons 1-4. 


Describe in writing the inter-relationships of the four 
major factors of HFE in a materiel development program 
i.e., human performance requirements, personnel selection 
criteria, training and equipment design-in accordance 
with lessons 1-4 and 28-29. 


Given the stages of the basic systems acquisition cycle, 
describe in your own words the level of involvement an 
HFE designer and evaluator should have at each point. 
Include in your discussion the purpose of your testing 
and the basic steps you would follow to become involved 
at each stage. Responses should be in accordance with 
lessons 21 and 22. . 


Discuss in writing the importance and purpose of evalua- 
ting human factors in a systems-.concept rather than just’ 
evaluating the operator or the machine individually in 
accordance with lessons 2, 21, and 22. 


Given an HFE proglem, analyze systems concept employing 
task analysis and/or determine the human factors require- 
ments which must be considered when designing the piece 
of equipment in accordance with lessons 6, 7, 14, and 
ei=24 . 

Given an HFE problem, identify 3 factors and 3 forces 
which would affect human performance and describe how you 
would measure each, i.e., what equipment to use and what 
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unit of measure 1S appropriate, in accordance with lessons 
eed, and 15-19. 


Discuss in writing the differences between HFE measure- 
ments taken in the field and those later in a laboratory 
setting. Include in your discussion the potential dif- 
ficulties of applying generalized laboratory data to 
field situations, in accordance with lessons 31 and 32. 


Discuss in writing the importance of performing testing 
both in a controlled environment and that in the ‘real 
world'"'. Responses are to be in accordance with lessons 
31 and 32. 


Given a list, identify the "experimental control" measures 
necessary in any test involving the human performance and 
describe the probable effects on the test data if they 

are absent, in accordance with lessons 31 and 32. 


Given an HFE problem with human performance data available 
calculate system performance reliability and assess the 
potential effectiveness of the system. Round your answer 
to the nearest thousandths. Responses are to be in 
accordance with lessons 21, 22, 26, 33 and 34. 


Given three different examples of a task analysis for the 
Same piece of equipment, select the one which is done 
correctly, in accordance with lessons 23 and 24. 


Given an HFE problem, determine performance measures for 
the dependent variables of time and. error in accordance 
With lessons 335 and 34. 


Describe the relationship between human factors engineer- 
ie ewana the engineering specialties of reliability, 
maintainability and safety in accordance with lessons 
eee). : : 


Given an HFE problem, state the HFE standards and speci- 
fications necessary to evaluate human performance in ac- 
cordance with the guidelines and reference documents 
Mmreaenced in lessons Zl and 22. 


Interpret the data given in an analysis of variance table 
performance scores from an experiment having three inde- 
pendent variables with one statistically significant 
first order interaction, in accordance with lessons 33 
and 34. 
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Table 23 


Recommended Resequencing of the Human Factors Engineering 
Self-Paced Course Terminal Learning Objectives 


Boome completion of the Human Factors Engineering Course, the 
student will demonstrate the following capabilities and 
knowledge: 


1. An understanding of common terms used in human factors 
engineering. 


2. <A familiarity with human factors references and an ability 
to use them. 


3. An awareness of potential sources of technical information 
on major human factors topics. 


17. An ability to interpret the standards and specifications 
of the human factors engineering community, through the 
mse Of HFE references. 


4. An understanding of the goals of human factors engineering 
in a materiel development program. 


5. A familiarity with the acquisition cycle. 


7. An understanding of the kinds of factors and forces which 
affect human performance and an ability to identify then. 


15. A familiarity with task analysis. 


7. An ability to measure the factors and forces which affect 
human performance. 


10; An understanding of basic statistical techniques, such as 
an analysis of variance. 


12. An ability to formulate performance measures for the de- 
pendent variables of time and error. 


11. An ability to calculate human performance reliability. 


9. An ability to integrate human factors principles in a 
DOD sponsored program. 


14. An understanding of the major techniques used by human 


factors specialist during system synthesis, design, and 
development. 
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An ability to determine human performance requirements 
in a systems concept. 


An ability to analyze human performance data within the 
context of "system effectiveness" and "system reliability". 


An awareness of the 
engineering and the 
maintainability and 


An awareness of the 
tory measurements. 


relationship between human factors 
engineering specialist of reliability, 
Siazety. 


differences between field and labora- 


An awareness of what "experimental control" measures are 
necessary for any test involving human performance and 
mie effects in their absence. 


An ability to apply 


the standards and specifications of 


the human factors engineering community. 
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elements of module (lesson) booklets and lesson topics: 

(1) Instructional materials are required to have 
a specific numbering system which clearly shows the relation- 
ships between terminal objectives, enabling objectives and 
comrse centent. 

(2) Each module booklet is required to have a 
cover page, module overview and lesson topics. ies ot 
the module overview and cover page are provided in Appendix I. 

(3) Lesson topics cover, wholly or in part, 
terminal objectives. Progress checks (self-tests) are pro- 
vided for the student at the end of lesson topics to measure 
their ability to achieve the terminal objectives and determine 
the need for remediation. 

(4) Each lesson topic shall contain the items 
listed below: 

Lesson Topic Cover Page* 
Lesson Topic Overview* 
List of Study Resources 
Lesson Topic Summary 
Narrative Form of Lesson Topic 
Programmed Instruction Material 
Lesson Topic Progress Check, complete with 
feedback and remediation. 
Samples of those items asterisk'd above are provided in 


Appendix J. 
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eo TroOrmat OL Course Lesson Books 

(1) Review "information frames" and questions to 
ensure that sufficient information has been provided to answer 
the questions before they are asked, i.e., Lesson 15, page 84, 
"frame from page 80"; Lesson 17, page 68, "frame from page 67". 

(2) Review "wrong answer frames", increase pro- 
fessionalism in replies and provide reinforcement by directing 
the student to a specific area to help choose the correct 
answer on the next try, i1.e., Lesson 12, page 68 ''(3) We 
know you don't know, but we want you to Be: ss a shot. Re- 
Mae to page 3539...". 

(3) Review the quality of the question asked. 
Rewrite those which are ambiguous, require guessing or really 
offer no doubt as to the answer because of the examples pro- 
vided in the wording of the answer choices, i.e., Lesson 13, 
mage 29, ‘frame from page 93". 

(4) Review the story line using the ficticious 
Bt. 1. M. Eager and if it.is decided to maintain the story 
line, upgrade the examples and remove extraneous portions 
which do not add to the content of the course. Provide more 
“real world" examples which could add greater meaning and 
eiarity to the content. 

(S) Change the "page flipping" format of the 
written version of the course. The U. S. Navy correspondence 


courses provide an acceptable format and one with which some 
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military personnel are already familiar. Another popular hoa - 


mat is used in the Elementary Algebra: Lecture Lab by Arthur 
H. Heywood. 


&. GOUrse Content. 

a. Review course content to be sure that the tasks 
identified in the learning objectives are adequately addressed, 
1.e., calculation of reliability. 

b. Correct clerical errors, i.e., Table 33.5 of 
Student Supplement; Lesson 20, page 74, says "frame from page 
71" when it should read "...page 24"; Lesson 19, page 91, frame 
fromepape 26: reads "By finding the point of ... between four 
hours...", Should read ''...two hours". 

4. Course Presentation. 

a. Provide student self-check tests following each 
lesson topic and following each module with a final comprehen- 
Sive test upon course completion (after the course review). 
The questions at the end of each information frame don't allow 
the student to adequately evaluate their understanding or 
acquisition of the necessary concepts and skills. Include 
reinforcement for end-of-information-frame tests by referring 
the student to the appropriate sections for review based on 
students’ incorrect answers. 

b. Provide course tests so that students can judge 
how well they had attained the course learning objectives be- 


fore they attempt to apply it in their normal jobs. This will 
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allow them to go back and review the areas in which they are 
weak. A certificate of completion and credit can then be 

awarded upon satisfactory completion of the course. Comple- 
tion of the course may also be made a part of each command's 


qualification procedures. 


5. Course Availability, Maintainability, and Supportabil- 

auey 

a. Provide plans and estimates required for future 
implementation of the HFE Course and an outline of the ne- 
cessary support in those areas as specified in NAVEDTRA 110A. 

b. Update this version of the course to be consistent 
with MIL-STD 1472C vice MIL-STD 1472B. When referencing 
material within any of the reference documents used within 
the course, do not use page numbers; provide section numbers 
instead. Addtionally, make references more specific when 
dealing with MIL-STD documents; do not just refer to the 
document as a whole when addressing a specific section. 

c. Recommend a schedule of periodic course review 


and updates as indicated by NAVEDTRA 110A. 


OS 
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VI. SUMMARY 


As this study indicates, the Human Factors Engineering 
Self-Paced Course has the strong potential for providing 
the stimuli to transfer knowledge and skills to its studénts. 
Whether this course provides sufficient content to effective- 
ly bridge the gap between the military technical documents 
and standards to allow for their increased usage by evalu- 
ators on the job is an area for further study. 

There is an interesting side note relative to the low 
level of human factors familiarity by all experiment parti- 
Cipants as indicated by the results on the pretest: could 
this possibly be substantiation to the claim made by GAO in 
Eiemmereport to the U. S. Congress [Ref. 1: pp. 1-27] which 


has been referenced periodically throughout this thesis? 
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APPENDIX A. 
RESOURCE DOCUMENTS SUMMARIES 


A brief summary of the various military standards, hand- 


books, and documents required for use by the Human Factors 


Engineering Self-Paced Course participants is provided 


within this Appendix: 


tis 
Dre 
Le 
4, 
a 
6: 


MIL-H-46855B 
MIL-STD-1472C 
MIL-STD-1474B 
MIL-HDBK-759 

Army Regulation 602-1 


Technical Memorandum 29-76 


Additonally, summaries or copies of the abstracts are 


provided for documents to which the Human Factors Engineering 


Self-Paced Course referred, but are not essential for course 


completion: 
ie Mi L-STD-721B 
Gee FED-STD=595 
3. Guide to Human Factors Engineering General Purpose 
Test Planning (GPTP) 
4. HFTEMAN 
29. HEDGE 
6. MOAT 
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REQUIRED DOCUMENTS 


1. MIL-H-46855B Human Engineering Requirements for Militar 
Systems, Equipments at Facilities 


"MIL-H-46855B establishes and defines the overall require- 
ments for applying the human engineering priciples and cri- 
teria presented in MIL-STD-1472C during the procurement of 
military systems, thereby effectively integrating man into 
the system. The specification requires the prospective con- 
tractor to state his approach in his Human Engineering Pro- 
gram Plan which is. submitted in response to the Request for 
Proposal RFP). The Human Engineering Program Plan, upon 
acceptance of hte proposal, becomes part of the procurement 
Gentract. 


The three major areas of the system acquisition process 
where these human factors engineering principles are to be 
employed are analysis, design and development, and test and 
evaluation. The objectives of analysis are identification 
and definition of system operations, maintenance, training 
and control functions; allocation of these functions to man 
and/or machine; analysis of the tasks comprising these func- 
tions; and development of system specific human engineering 
design criteria and operation and maintenance procedures. 
The human engineering inputs developed in the analysis phase 
and comensurate with MIL-STD-1472C are to be incorporated 
into the detailed design. These human engineering provisions 
Shall be evaluated during the design reviews to ensure their 
adequacy. The purpose of the test and evaluation phase is 
assurance of the fulfillment of MIL-H-46855B and contract 
requirement, demonstration of conformance to MIL-STD-1472C 
quantification of man-machine system performance, and indi- 
cation of the introduction of possible undesirable design or 
procedural features. 


‘MIL-STD-1472C and MIL-H-46855B are complementary direc- 
tives that deal with requirements for implementation of human 
PeecOrs engineering principles in the development and acqui- 
Sition of military systems, equipment and facilities. The 
former establishes the criteria by which the systems are to 
be human engineered, and the latter establishes the require- 
ments for maintenance and maintainability as integral parts 
of the human engineering of the total system.” 
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@. MWMIL-STD-1472C. Human Engineering veo corinceria for 
Military Systems, Equipments and Facilities. 


"This standard establishes general human engineering cri- 
teria for development of military systems, subsystems, equip- 
ment and facilities. 


The purpose of this standard is to present human engineering 
design criteria, principles, and practices to achieve mission 
success through integration of the human into the systen, 
subsystem, equipment, and facility, and achieve effectiveness, 
Simplicity, efficiency, reliability, and safety of system 
operation, training, and maintenance. 


More specifically, its purpose is to present human engineer- 
ing design, criteria, priciples and practices to be applied in 
the design of systems, equipment and facilities so as to: 

a. Achieve required performance by operator, control and 
maintenance personnel. 

b. Minimize skill and personnel requirements and training 
time . 

¢. Achieve required reliability of personnel-equipment 
combinations. 

d. Foster design standardization within and among systems. 


The standard includes a compilation of anthropometric data 
Peomeseveral military sources (MIL-STD-721B, MIL-STD-1474B 
and FED-STD-595), an extensive collection of control/display 
design criteria, hazard and safety considerations, and require- 
ments for certain specialized systems which may also be applied 
to such equipments as ground vehicles, remote handling devices, 
air and ship crew stations," 


3. MIL-STD-1474B Noise Limits for Army Materiel 


Piaree distinctly different types of ‘noise criteria' 
Whieh are used to limit noise exposure have evolved over the 
years. It 1s important to distinguish among the three types 
in order that the proper type may be chosen for application and 
use in various situations. The three types of noise criteria 
are: 

a. Hearing damage-risk criteria 

b. Hearing conservative criteria 

c. Materiel design standards 


This document is a design standard for noise. It is based 
On provisions of TB-MED-251 with respect to noise exposure 
Gmitemeae MIL-STD-1472C extracts communications criteria from 
i 
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4. MIL-HDBK-759. Human Factors Engineering Design for Army 


Materiel. 


"This: handbook gives the design engineer both human factors 
engineering design priciples and detailed criteria. The design 
principles are stated as general rules to be applied during 
system-development programs or.as essential items that must be 
considered during design to insure that- sound human factors 
engineering practices will be'incorporated. The detailed cri- 
teria consist of dimensions, ranges, tolerances, and other 
Specific data. In some cases, the range of acceptable dimen- 
Sions and other factors may be rather large. Where only the 
minimum and maximum are given, design engineers may select any 
part or item within the recommended range. But where optimum 
dimensions are given, designers should aim to approximate them 
whenever possible. 


The purpose of this document is to establish in handbook 
form general data and detailed criteria for human factors 
engineering application in the design and development of army 
maceriel . 


The information in this handbook is a consolidation of 
the material contained in four U. S. Army Human Engineering 
Laboratory Standard Documents; HEL S-2-64A, S-3-65, S-6-66, 
and S-7-68.. The user will also note the similarities of this 
document with MIL-STD-1472C." : 


>». Army Regulation 602-1. Personnel-Materiel Systems: Human 
Factors Engineering’ Program. Washington, D. C., June 1976. 
"This regulation prescribes policies and procedures and 

assigns responsibilities for human factors engineering (HFE) 

in the Department of the Army. For the purpose of this reg- 

ulation, -HFE is defined as a.comprehensive technical effort 

to integrate all personnel characteristics (skills, training. 

implications, behavioral reactions, human performance, 

anthropometric data and biomedical factors) into Army doctrine 


and systems to assure operational effectiveness, safety, and 
freedom from health hazards." 


6. TECHNICAL MEMORANDUM 29-76. Guide for Obtaining and An- 
alyzing Human Performance Data in a Materiel Development 
POO . 


"The specific objectives of this report are to: (1) de- 
scribe how to conduct and report an HFE test according to the 
requirements of DI-H-1334A, (2) detail the expenditures in 
time and money associated with the conduct of an HFE test, 
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(3) provide examples of HFE test reports for systems in "ex- 
perimental prototype" and "advanced development" phases of _ 
development, (4) describe the uses of the obtained HFE test 
data as a function of system development, and (5) explain the 
impact of the DI-H-1334A findings on a materiel development 
program. 


This document is written for government contract monitors, 
contract project directors, and contractor HFE personnel. 
The guidelines for conducting and reporting on the HFE test are 
intended for experienced HFE personnel. 


Questions of what data are to be collected, how they are to 
be collected, and how the data can be used are discussed in 
Gmepters 2, 3, and 4 of this report. Chapter 2 is a guideline 
for planning, conducting, analyzing, and reporting on an HFE 
evaluation according to the requirements of DI-H-1334A. Pro- 
cedures for managing the HFE evaluation, allocating test 
personnel; developing test cost estimates, etc., are also con- 
tained in Chapter 2. This information will aid project man- 
agers in the administration and organization of an HFE 
evaluation. The explanations of the DI-H-1334A:requirements 
will assist the contract monitor to understand and monitor HFE 
tests and ensure that all of the requirements of DI-H-1334A are 
satisfied. 


Chapters 3 and 4 supplement the guidelines given in Chapter 
2 by giving detailed examples of HFE reports, written according 
to the requirements of DI-H-1334A. Chapter 3 presents the 
HFE test report of a system.in the experimental prototype 
Stage of development. This sample HFE report focuses on de- 
terminining the feasibility of human performance, the appro- 
priateness of the tasks allocated to the operator and to the 
machine, and the adequacy of the workspace layout. This report 
also demonstrates procedures for conducting HFE tests and mock- 
up equipment. 


Chapter 4 contains the HFE test report of an advanced 
development prototype. The emphasis in this stage of develop- 
ment 1S on determining the capability of the operator to 
perform his assigned tasks within his prescribed time and 
error standards. This test report also evaluates the adequacy 
of operator selection and training, as well as the equipment 
configuration. 


Chapter 5 discusses implications of human performance tests. 
The uses to which data can be applied and the problem associ- 
ated with conducting HFE tests are also described. By de- 
scribing how HFE test data can be applied and the benefits 
of collecting the data, program managers can better appreciate 
tae meed for HFE testing." 
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RECOMMENDED DOCUMENTS 


1. MIL-STD-721B Definitions of Effectiveness Terms for 
Reliability, Maintainability, Human Factors and Safety 


"Applying effectiveness terminology within the Department 
of Defense and industry requires good communications and 
coordination which, in turn, requires common definitions and 
terminology. This Standard lists and defines words and terms 
most commonly used in reliability, maintainability, human 
factors and safety. Statistical and mathematical terms which 
have gained wide acceptance are not defined in this Standard 
Since they are included in other documents, e.g., MIL-HDBK-217. 


The following criteria were used for the inclusion of 
terms and definitions deemed pertinent to the scope of this 
Standard: 

a. Terms and their definitions which are: 


(1) Important in procurement of weapon systems for 
precise definition of effectiveness criteria. 


(2) Unique in their definition, allowing no other 
meaning. 


(3) Expressed clearly, preferably without mathematical 
symbols. | 


(4) Fully explanatory, without reference to any other 
source documents. 


b. Terms that were avoided include: 
(1) Those found in ordinary technical, statistical, 
or standard dictionary or text Haves a singularly acceptable 


meaning when used in the context. 


(2) Terms already existing in other Military Standards 
Gutside of the project scope. 


(3) Multiple word terms, unless needed for uniqueness. 
c. The purpose of this Military Standard is to standardize 


On meanings of terms for the particular application, not to 
compile a handbook. 








eee D stD 595 Colors 


"The purpose of this standard is to present in covenient 
form a collection of standard colors currently used by the 
various departments of'the Government. These standard colors, 
identified by 5-digit numbers, are defined by fundamental 
colorimetric data. For reference purposes, each color is 
reproduced within a 1/2 x 1 inch chip. For procurement, in- 
spection and other color matching purposes, 3 x 5 inch color 
chips are issued as specific standards. These chips are 
identified by the same 5-digit numbers. 


Spectrophotometric curves and Commission Internationale 
de l'Eclairage (CIE) data for each chip are included as 
Volume II to this standard. These are to be used as basic 
standards for reproducing future issues of the chips, also 
for determining changes which may, or may not, have occurred 
iieenips in stock. The spectrophotometric curves and CIE 
measurements may also be used for acceptance testing purposes 
Tiemeteer the 3 x 5 inch chips if so specified.. However, 
accurate comparison can be made only if values and curves 
are obtained on the same instrument standardized under the 
Same conditions." 


3. Guide to Human Factors Engineering General Purpose Test 
Planning (GPTP) 


"This report is concerned with human factors engineering 
test and evaluation program planning, which is generalizable 
across naval weapons systems except for nuclear weapons and 
propulsion subsystems. The Test and Evaluation Program 
Planning Guide is prepared for use by test planners and’ is in 
accordance with DOD Directive 5000.3, NAVMATINST 3900.0, 
NAVMATINST 3960.6, OPNAVINST 3960.10, and OPNAVINST 4720.9. 
The human factors engineering Test and Evaluation Program 
Plan specifically addresses developmental and operational 
tests as well as production acceptance tests to provide data 
for satisfying the Defense System Acquisition Review Council 
milestones for system acquisition." 


4. HFTEMAN Human Factors Test § Evaluation Manual, Vols. I, 
iee, vata 1d 


"In most military systems the ultimate effectiveness of 
the system depends on the capability of the human to operate, 
maintain, transport, erect, or otherwise use the system equip- 
ment. 
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In the test and evaluation of such systems adequate con- 
- sideration must be given to the human element. 


Human Factors Test & Evaluation Manual was developed to 
assist Navy and Marine Corps Test and Evaluation personnel in 
 _ the evaluation of the human factors aspects of equipment items. 


Human Factors Test §& Evaluation Manual is primarily di- 
rected toward the Human Factors Engineering (HFE). The detail 
provided for the Test and Evaluation Plan: 


-- Guidelines concerning what to evaluate. 
-- Standards or criteria to evaluate against. 


-- Planning information on how to design, setup, conduct, 
and analyze data from a human factors enginering test 
or evaluation. 


Human Factors Test § Evaluation Manual is therefore used 
when the detail Test and Evaluation plan is being generated. 
Meeetracles the planner to identify HFE: 


-- Test objectives. 
-- Test methods, procedures and conditions. 


-- Test measurements to be acquired and recorded during 
the test or evaluation. 


-- Criteria or standards against which measurements are 
compared. 


-- Test data collection materials such as questionnaires 
om cheekilists:. 


-- Test data analysis, presentation and reporting require- 
ments. 


Human Factors Test § Evaluation Manual consists of three 
volumes: 


=- Volume I - Data Guide: contains the guidelines concern- 
ing what to evaluate for different classes 
of equipment and types of tests. 


a volume Ii- Support Data: contains additional criteria 
expanding on the guidelines contained in 
Volume I. 


-- Volume III-Methods and Procedures: contains guidance on 
how to design, set up, conduct and analyze 
data obtained in HFE Test & Evaluation program. 
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5. Human Factors Engineering Data Guide for Evaluation (HEDGE) 


"The purpose of the information in HEDGE is to enable you 
to expand your test capabilities in considering the human 
element. They will provide you with a strategy for viewing 
any item which is undergoing testing from the standpoint of 
the soldier who must ultimately operate, maintain, or other- 
wise utilize it. The use of these materials, in addition to 
Standard Task and Design Checklists and Questionnaires, will 
enable you to tailor your HFE subtest to a specific item. 


These materials have been prepared especially for you: 


a. They are intended to support test engineers not design 
EmeIneers. 


b. They were designed with your specific tasks in mind, 
1.€., preparing a Test Plan, conducting a test, analyzing 
and interpreting test data, and generating the test report. 


Because these materials offer you a strategy for conducting 
human factors testing rather than a compendium of facts, the 
results which you obtain will be directly proportional to 
your intelligent and common sense application of the data 
Presented." 


6. Mission Operability Assessment Technique: A System 
Evaluation Methodology (MOAT) 


"The MOAT is an evaluation methodology that measures the 
operability of a system or subsystem in terms of operator 
tasks performed during a mission. MOAT addresses the problem 
of how well an operator can use a system or subsystem to per- 
form tasks within the mission context. Contrasted to evalu- 
ations using human engineering design criteria which present 
only pass or fail information, this technique provides infor- 
mation on the degree of system and/or subsystem success or 
failure. This report discusses how MOAT was developed, how 
MOAT can be used on existing or emerging systems, and how MOAT 
will be expanded in future development to include multicrew 
Station evaluations. This report examines how the three under- 
lying technologies have been integrated into one comprehen- 
Sive methodology. Task analysis, scaling methodology, and 
multi-attribute utility theory are-discussed in terms of MOAT 
development and application. Finally, the report discusses 
the importance of developing systems evaluation methodologies 
which provide decision makers with meaningful information 
necessary for effective decision making." 
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APPENDIX B. 


HUMAN FACTORS ENGINEERING SELF-PACED COURSE 
TASK REQUIREMENTS 


Upon completion of the HFE Course, students should know: 


i. 


oe 


0 . 


el. 


ihe meaning of common terms used in human factors engi ~ 
neering. 


MWe principal HFE references (regulations, standards, 
specification, guides, etc.) and where’ in them to look 
for the answers to specific types of HFE problems. 


Potential sources of technical information on major topics 
within human factors engineering. 


The goals of HFE in a materiel development program (i.e., 
the compatibility of the four factors: human performance 


requirements, personnel selection criteria, training and 


equipment design). 


When, how, and for what purpose in the general scheme by 
which the Department of Defense (DOD) develops materiel 
human factors engineering activities should be performed. 


How to determine the human performance requirements in a 
systems concept. 


‘The kinds of factors and forces which can affect human 


performance and how to identify and measure them. 


The differences between measurements taken in field vs. 


-laboratory settings, and the difficulties of HORE 2a 


laboratory Bene to field Situations. 


How to Eanes and then state in comprehensible English 
performance measures applicable to a specific system for 
the general dependent variables of time and error. 


What "experimental control" measures are necessary in any 
test involving human performance and the probable effects 
on test data if they are absent. 


How to analyze human performance data (e.g., time and 
error) within the context of "system effectiveness" and 
"System reliability". 





ae 
iS. 


14. 
iD. 


16% 


ag er 


HS . 


ag. 


How to calculate human performance reliability. 


The major techniques used by human factors engineers 
during system sythesis, design and development. 


How to recognize a good task analysis. 


The relationship between human factors engineering and 
the engineering specialities of reliability, maintain- 
ability and safety. 


How to interpret and apply the standards and specifica- 
tions needed for human factors engineering that deal 
with human performance. 


The correct method of stating criteria for a vehicle 
which must be inaudible 100 meters away from it, and 
for an alarm which must be audible for 500 meters. 


How to determine the minimum height of letters on a 
Sign which must be visible at 100 meters on a sunny day. 


How to interpret an analysis of variance table of per- 
formance scores in an experiment having three indepen- 
dent variables with one statistically significant main 
effect and one statistically significant first-order 
pateraction. 
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Meee NDIX C. 


HUMAN FACTORS ENGINEERING SELF-PACED COURSE 
TERMINAL LEARNING OBJECTIVES 


Upon completion of the Human Factors Engineering Self-Paced 
Course, the student will demonstrate the following capabili- 
ties and knowledge: 


it. 


10. 


es 


aPZ 


aS. 


An understanding of common terms used in human factors 
engineering. 


A familiarity with human factors references and an ability 
to use then. 


An awareness of potential sources of technical information 
on major human factors topics. 


An understanding of the goals of human factros engineering 
in a materiel development program. 


An ability to integrate human factors priciples in a DOD 
sponsored program. 


An ability to determine human performance requirements 
in a systems concept. 


An understanding of the kinds of factors and forces which 
affect human performance and an ability to identify and 
measure them. 


An awareness of the PESOS between field and labora- 
tory measurements. 


An awareness of what "experimental control" measures are 
necessary for any test involving human performance and 
the effects in their absence. 


An understanding of basic statistical techniques, such 
as an analysis of variance. 


An ability to calculate human performance reliability. 


An ability to formulate performance measures for the de- 
pendent variables of time and error. 


An ability to analyze human performance data within the 
context of "system effectiveness" and "system reliability". 
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14. 


5. 
HAD”. 


ie 


An understanding of the major techniques used by human 
factors specialists during system synthesis, design, and 
development. 


A familiarity with task analysis. 

An awareness of the relationship between human factors 
engineering and the engineering specialists of reliability, 
maintainability, and safety. 


An ability to interpret and apply the standards and speci- 
fications of the human factors engineering community. 





APPENDIX D. 
HUMAN FACTORS ENGINEERING SELF-PACED COURSE 
LESSON TOPICS OUTLINES® 
Lesson 1: Welcome to Human Factors Engineering 
I. Course Introduction 


II. Importance of Human Factors 


Lesson 2: Why Human Factors? 


im otory Line 
A. Introduction to main characters 
Peis M, Lager 
Zo Cape. B. Smart 


II. Human Factors Engineering 
A. Definition of 
B. Historical Perspective 
mii. systems 
A. Man-machine Systems 
B. Components of Man-machine Systems 


Lesson 3: Tragic Mistakes and Positive Consequences 


I. Historical Perspective 
A. Examples 


II. Common Errors - Reading and Interpreting Instruments 
IiI. Common Errors - Operating Controls 
we cipmaeme Seatus of Human Error 


V. Man-Machine Incompatibilities 
A. Man-machine Capability Differences 


SExtracted directly from the Student Supplement of the 


Human Factors Engineering Self-Paced Course. 








Lesson 4: Basic Information Processing, or Is Man a Machine? 


I. Attention 
x. Liners to 
B. Selective Attention 


II. Judgments 
A. Absolute 
B. Relative 


III. Man-machine Systems. 
A. Man the Processor 
B. Stimulus-Response Codes 
C. Information Transmission 
D. iInput-output Processes 


Lesson 5: History and Related Technology, or Human Factors, 
iiirseis Your Life 


I. History - HFE 
A. Ancient Cultures 
B. Industrial Revolution 
C. World War II 
ee “Current Efforts 


e 


II. Stage of HFE Development 


A. Pretechnology 
B. Aerospace 

c. wsoevotechnical 
D. Cosmopolitan 


III. Review 


Lesson 6: Anthropometry, or Do I Fit? 


i. introeucti on. 
A. Definition 
B. Historical Perspective 


II. Design Principles 


A. Adjustability 
B. Sth - 95th Percentile Range 
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Lesson 7: Work Space Design and Arrangement, or Don't Cramp 


MymSty le 


Work Space Envelope 
A. General Design Principles 


Analytical Methods 


A. Indexing/Rating Method 
B. Link Analysis 
C. Proetetypes 


Component Location and Spacing Problems 


Lesson 8: Vision Capabilities, or A Shot In The Dark! 


Ie 
ied. 
1 J 


Ie 


Visual Anatomy-Supplement 
Visibility 

Panel Lighting 

A. Flood Lighting 

B. Integral Lighting 


Visual Detection, Identification, and Estimation 


Lesson 9: Vision Displays, or Are My Eyes Deceiving Me? 


ee 


oe. 


Types of Displays 
A. Quantitative 
B. Qualitative 
Cc. Siatac 

D. Dynamic 


Scales 

A. Digital 

B. Fixed Pointer 
C. Moving Pointer 


Signal and Warning Lights 
A. Detection Factors 


Alphanumeric and Symbolic Characters 


ZS 





LESSON 10: Auditory Presentations, or When Is An Alarm Not 
An Alarm? 3 ; 


I. General Types of Displays 

A. Displays Using Auditory Modalities 
B. Nature of the Message 

II. Sound Characteristics 
ie dB, Hz. 
B. Frequency, Intensity 
C. Human Reception Capabilities 

meee General Display Principles 

IV. Presentation Principles 


V. Warning and Alarm Systems 


LESSON 11: Standardization of Controls, or Which Way Is Up? 
I. Standard Arrangement - Population Stereotypes 


II. Categorizing Controls 
A. Quantitative, Qualitative, Representational, and 
Continuous 


III. Functions of Controls 
A. Continuous Adjustment Settings 
B. Discrete Settings 


ms lypes of Controls 
A. General Types - Linear, Rotary 
B. Specific - Pushbutton, Toggle Switch, Knobs, etc. 
ec encral Raves for’ Selection ; 
eee olD-1472C and MIL-HDBK-759 
LESSON 12: .Positioning of Controls, or The Right Place At The R 
Right Time 


ls Becidental Activation of Controls 
A. Prevention Methods 
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tele. 


Location of Controls and Displays 

A. Priority, grouping association factors 
B. Location: and body position 

C. Spacing between controls 


LESSON 13: Control Dynamics, or How Hard To Twist The Knob 


we 


Hel. 


it. 


Design Concepts 
A. S-R compatibility 


B. Control-display ratio, a type of compatibility 


Types of resistance 

A. Static friction 

Pe olaaame f£riction 
i. clastic resistance 
D. Viscous damping 


Feedback 
oe «CCU It rinsic 
Bm. Extrimeeac 


LESSON 14: Other Senses, or Controls That Have Shaped-Up 


ae 


Review lessons 11-13 


Touch 
A. Pressure Sensitivity 
B. Pain 


C. Temperature 


Touch Coding 

A. Shape coding 

B. Class ‘A’ and 'B' design 
fo texture coding 


LESSON 15: Vibration and Acceleration, or Take It Easy I Have 


i. 


ae 


Weak Stomach! 


Vibration 

A. Definitions 

B. Measurement 

C. Body Parts affected 


Whole Body Vibrations 
A. Performance effects 
B. Tolerable limits 








ee 


Acceleration 
A. Performance effects 
C. Protective measures 


LESSON 16: Vigilance, or Stay Awake If You Can 
me Vigilance 


A. Definition 
B. Examples 


II. Performance Decrements Due to Vigilance 
A. Time Frame 
B. Magnitude of decrement 
C. Display characteristics 
Bt Signal Characteristics 
A. Noise 
Bb. signal intensity 
C. Rate of signal presentation 
IV. Other Factors Affecting Watch-keeping Behavior 
A. Environmental conditions, noise 
B. Atmospheric temperature 
C. Procedural conditions, work/rest schedules 
LESSON 17: Temperature Effects, or Baby, It's Cold Outside 
ie Reactions to Temperature Change 
A. Physiological 
B. Adaptation, sensitization and habituation 
ee eteeecctive temperature 
fee Extreme Cold 
A. Physiological and performance effects 
BESSON 18: Atmospheric Effects or I Can't Breathe 
I. Description of Atmosphere 
A. Gases, density, pressure 
i. Hypoxia 
A. Factors causing 
B. Effects - physiological and performance 
et. CO. and CO 
A Physiological and performance effects 


B. Military specificiations 
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BV. 


Ve 


Radiation 
A. Measurement Definitions 
B. Effects of overexposure 


Prevention 
A. Mask 
B. Clothing 


LESSON 19: Noise, or Can You Hear Me? 


Auditory Anatomy 

A. Structure 

B. How sound travels in the ear 

C. Mechanical and electrical transmission 


Properties of Auditory Stimuli 
MM Frequency, imtensity 


Communication 
A. Maskers 


Physiological Effects 
=» ITS 
B. PIS 
C. Damage risk criteria 


Protection Against Noise 


LESSON 20: Review of Lessons 1-19 


LESSON 21: System Acquisition 


ies 


eT. 


Department of Defense Acquisition Policy 
A. Variation across systems 

B. OMB A-109 

e. bop sv00.l1 


Phases of Weapons Systems Acquisition Process 
A. Concept exploration 

B. Demonstration and validation 

C. Full-scale development 

D. Production and deployment 


HFE and the Acquisition Process 
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LESSON 22: System Analysis, or The Big Picture 


lie 


Pele 


Purpose of System Analysis 

A. Scheduling 

B. Identifying limiting factors 
c.) oyotcm Perloumance criteria 
D. Design Options 

E. Evalutation of Systems 


Stages of System Analysis 
A. Requirements analysis 
B. Function analysis 

C. Task analysis 


Major Problem Areas 
A. Subsystems! interaction 
1. Sectionalization technique 
B. Criterion determination 
C. Defining human performance effectiveness 


LESSON 23: Task Analysis: History and Perspectives 


Te 
ii. 
mi. 


IV. 


. 
| 


Recent History of Task 

Definition of Task Analysis 

Other Improtant Task Analysis Factors: 
A. Systems mission and function 

B. Job, task, sub-task, task element 
C. Task inventory, task taxonomy 


Output of Task Analysis 


A Design 

B. Training 

C. Test and evaluation 
D. Manning 

E. Workload 


m™ LESSON 24: Task Analysis, or Fitting Task Analysis into the 


1 


ele. 


system 


Stages in Task Analysis Process 

A. Identify task, sub-task, task element 

B. Develop specific behavioral objectives 
C. Identify supporting skills and knowledge 


Task Statements 
A. Specific behavioral objectives 
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III. Task Analysis Worksheet 


IV. Sampling Techniques 
A. Activity sampling 
B. Process Analysis 


LESSON 25: Affordability, or When Can We Trade Off What? 


I. Trade-off analysis 
A. Definition 
Ee. «ypes. 
1. Geometry of design 
2. Manpower allocation 


II. General System Analysis 
A. Operational requirement 
B. Hardware design 
C. Manpower and training requirements 
D. Saftey, reliability,and other factors 


III. Four Major Steps in Trade-off Analysis 
A. Baseline Alternative 
ic Lee 
2. Initial Hardware 
5. Manpower and training requirements 


IV. Life-cycle Costs 
A. Computer models 


LESSON 26: Maintainability, or Can Anybody Fix This? 


Paebasic Concepts 
A. Maintainability 
B. Maintenance 
C. Redmability 
De Human performance 


II. Design Features 

Pt. okilleAppiication 
A. Capabilities/limitations 
B. Training 


IV. Predictions 





LESSON 27: Hazard Analysis, or A Stitch in Time 


i, 


iy 


We 


Introduction 
A. HFE and safety 
B. HPE and safety design 


Introduction to Hazard Analysis 


Identification Phase 

A. Checklists 

B. Historical records - intermediate indicators 
C. General Investigations 


Evaluation Phase 
A. Grouping according to category 
B. Ranking within category 


Cost Countermeasures 


LESSON 28: Training the Right People 


ize 


el. 


Selection and Training in Design/Conceptual Phase 
A. Approaches 

1. Organizational Analysis 

2. Job analysis 

53. Person analysis 


Training Techniques 

On-the-job training 
Classroom training 
Computer-aided instruction 
Team Training 


DOW D> 


LESSON 29: Does Training Work ? 


lige 


JTeib% 


ISD Phases 

A. Analysis 

B. Design 

CC. Pexctlop 

D. tmplement 

Bo. J/Gorerol 

Validity and Reliability 
A. Types of validity 


Je Tlbrce eres g at eesy 

Z reonsormance 

Bo) intra-Organizational 
4 Inter-organizational 
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III. Simulator Training Issues 


mepoON 30: Psychophysical Methods, or Do I Detect a Signal? 


I. Psychophysics 
A. Definition 
Bp. Pugpose 


II. Classical Methods 
A. Methods of limits, adjustment and constant stimuli 
B. Assumptions 
C. Advantages, disadvantages 


mee  lheemy of Signal Detection 
A. Response bias 
B. Decision matrix 


IV. Scaling Methods 
A. Direct 
B. Indameet 


LESSON 31: Experimental Methods, or How Do I Know If I've 
Done It Right 


I. Eager's Experiment 


Ii. Researeh Categorization 
A. Theoretical 
B. Empirical 
1. Observation, correlation, experimenttation 
2. Natural (field), laboratory 


III. Variable Classifications 
A. Qualitative and quantative 
B. Independent and dependent 
C. Relevant 
ly Sup ect 
oe Soleuational 
53. “Sequence 


LESSON 32: Experimental Methods, or How Do I Control These 
Influences? 


I. Control Techniques 
A. Subject 
B. Sitwattional 
C. Sequence 
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II. Designs 
A. Within subject 
B. Between subjects 
1. Simple randomized 
2. *Faetvor pal 


III. Validity 
A. Internal 
B. External 


Lesson 33: Statistics, Part I 


is introduction 
A. Description of total statistics lessons 
Pen §DEsScription of statistics 
C. Purpose of statistics 


II. Frequency Distributions 


A. Ungrouped data 

m. internal data 

C. Meaning of any one score 
D. Percentile ranks 


met. 8Measures of Central Tendency 
A. Mean 
B. Normal curve 
C. Median 


IV. Measures of Despersion 

Mean deviation 

Standard deviation 

Variance 

Relation of 50 and normal curve 


0TOW YS 


EeEoOON 54: Statistics, Part II 


iercorrelation 
ae DemEreec 
B. Direction 


II. Inferential Statistics 
A. Nonparametric 
B.  Parawetric 
C. Appropriate statistical tests 


1. t-test 
2. F test 
53. ANOVA 
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mist) Intewpmetation of Results 


A. Practical vs. statistical significance 
B One-way ANOVA 

C. Two-way ANOVA 

D Higher order ANOVAS 


LESSON 35: Review, or How Have I Done So Far? 


LESSON 36: Human Factors and the Military 

oF AR 602-1 

ie. MIL-STD-1472C 

ier. MIL-HDBK-759 

iy. MIL-STD-1474B 

LESSON 37: Human Factors Test and Evaluation, or Can a 
HFTEMAN Cross a MOAT using a HEDGE? 
ls HF Testing 

A. HEL TM 29-76 
B MOAT 
C. HFTEMAN 
D. HEDGE 
Ee HREES 
H 
A. 


tel. uman Performance Measures 


Analysis of human performance data 
LESSON 38: Real World Problem, Part I 
LESSON 39: Real World Problem, Part II 


LESSON 40: Overall Summary 


pay 





APPENDIX E 


EVALUATION PLAN FOR THE 
HUMAN FACTORS ENGINEERING SELF-PACED COURSE 
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Table of Contents 


Acronyms and Abbreviations 


References 


Bibliography 


Section I - Introduction to the Plan 


Section II - Administrative Information 


Section III - Scope of the Evaluation 


Section IV - Evaluation Guideline 


Section V 


Annex A 


Annex 
Annex 
Annex 
Annex 
Annex 
Annex 


B 


C 
D 
E 
IS 
G 


Report(s) 
- Student Information Forms 


- CCS Log Format 


Diagnostic Pretest 


Diagnostic Post Test 


CCS Questionnaire 


Student Attitude Questionnaire 


Course Objectives Questionnaire 
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tobe 


tele 
pies 
IV-1 

cod E 





COTF 
ccs 
DCOTF 


HFE 
HFTEMAN 
NPS 
PMTC 
SIF 


Acronyms and Abbreviations 


Commander Operational Test and Evaluation Force 
Command Course Supervisor 


Deputy Commander Operational Test and Evaluation 
Force 


Human Factors 

Human Factors Engineering 

Human Factors Test and Evaluation Manual 
Naval Postgraduate School 

Pacific Missile Test Center 


Student Information Form 





(a) 


(bd) 


(cr) 
(d) 


(e) 


=) 


(g) 


(h) 


(i) 


(a) 
(ic) 


References’ 


Effectiveness of U, S, Forces Can Be Increased Through 


Improved Weapon System Design; Comptroller General 
Report to the Congress of the U. S.; January 29, 1981; 


PSAD-81-17 


Human Factors Test and Evaluation Manual (HFTEMAN), 
Vols. I, II, and III, Octover 1976, Pacific Missile 
Test Center 


OPNAVINST 3960.10A, July 31, 1980 


Human Factors Engineering Self-Paced Course - Student 
Supplement and Lessons 1-40, Pacific Missile Test Center 
Pt. Mugu, California (Preliminary Draft) 


Interservice Procedures for Instructional Systems De- 
mer Phase I: Analyze, 1 August 1975, NAVEDTRA 
1O6A 


Interservice Procedures for Instructional Systems 
oe: Phase II: Design, 1 August 1975, NAVEDTRA 
LO6A 


Interservice Procedures for Instructional Systems 
—o Phase III: Develop, 1 August 1975, NAVEDTRA 
LOGA 


Interservice Procedures for Instructional Systems 
Development, Phases IV & V: Implement & Control, l 
August 1975, NAVEDTRA 106A 


Interservice Procedures for Instructional Systems 
Development, Executive Summary and Model, 1 August 1975, 
NAVEDTRA 106A 

COMOPTEVFORINST 3960.1B 5 July 1979 


Procedures for Instructional Systems Development, 
12 July 1978, NAVEDTRA 110 


"Several of the references used to design the test plan 


have been revised and incorporated into the main body of the 
thesis. The criteria and standards are in accordance with 
the updated versions of the above instructions. 
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Section I 


Introduction to the Evaluation Plan 


101. Purpose. The purpose of this evaluation plan is to 
assess the operational effectiveness of the Human Factors 
Engineering (HFE) Self-Paced Course and its potential value 


for use in the military test and evaluation community. 


102. Course Description. This course is divided into three 
major sections, Section I (Lessons 1-5) deals with the 
human's capabilities and limitations. In this section, the .. 
history and continued need for human factors engineering will 
be investigated. Lessons 6-10 are concerned with the physio- 
logical capabilities and limitations of human beings. Lessons 
11-13 discuss how the proper design of controls and displays 
makes use of the information learned in the previous lessons. 
Lessons 14-19 take a look at human interactions with the 
environment, followed by a review of the first section in 
lesson 20, 

The primary focus of Section II is upon the human fit- 
ting into the system. Lessons 21-27, investigate the role of 
the human engineer in various aspects of analysis, such as 
systems analysis, cost analysis, task analysis, etc. Lessons 
28-29 are concerned with the selection and training of person- 
nel. In lessons 30-34, the student will receive a short course 
on experimental techniques and statistical concepts. Finally, 


lesson 35, will offer a review of Section II. 
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Section III is entitled "Human Factors in the Military". 
In this section, lessons 36 and 37 focus on human factors 
organizations, documentation, and future application. 

In order to give the students a practical application of 
what has been presented in the course, lessons 38 and 39 will 
ask him/her to work on a "real world" problem. Finally, in 
lesson 40, a more typical review of the entire course is 


presented. 


103. Background. The HFE Self-Paced Course was developed to 
satisfy a need for increased awareness and more indepth under- 
standing of Human Factors. 

Reference (a), highlights this need and identifies a 
deficiency in the performance of various weapons systems 
“,...-because the DOD does not pay enough attention to logistic 
support, human factors and quality assurance during the design 
phase of the acquisition process. These problems deter the 
systems’ effectiveness to defend our country in case of war." 
The Government Account Office (GAO) “therefore makes recommen- 
dations to improve the management and planning of ownership 
considerations that have an impact on the effectiveness of a 
weapon systen." 

The Human Factors Test and Evaluation Manual (HFTEMAN), 
Vols. I, II and III (reference (d)) was distributed to various 
government agencies in October 1976 by Pacific Missile Test 


Center, Pt. Mugu, California. HFTEMAN was developed to provide 
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standardization in procedures, testing and criteria in eval- 
uating human factors. It, however, assumes a basic knowledge 


of human factors for its most effective use. 


The HFE Self-Instruction course was developed in order to 
provide this basic knowledge. It has not been evaluated or 
used on a trial basis in any portion of the military prior to 


this time. 


141 I-3 





Section II 


Administrative Information 


201. General. General responsibilities for activities in- 
volved in this evaluation are provided in this section, as well 
as appropriate points of contact. Continuing close liaison is 
essential to timely and successful prosecution of this evalu- 


ation. 


202. Responsibilities. 
a. Naval Postgraduate School (Lt. M. M. Fleming) 


(1) Promulgate major changes to this evaluation plan. 

(2) Coordinate arrangements for HFE course subjects. 

(3) Coordinate distribution of required course 
materials to participants. 

(4) Conduct briefings for all participating if so 
requested and funded by the requesting activity. 

(5) Provide certificates of completion or equivalent. 

(6) Analyze evaluation results and make them avail- 


able to the appropriate units upon request. 


b. COMOPTEVFOR, DEPCOMOPTEVFOR, AFT&E Center/TELH, TESTG/ 
ENAH 
(1) Purnish names and/or Student Information Form 
(SIF) numbers of participating students. 
(2) Provide point of contact for test subjects with- 
in each command (Command Course Supervisor - CCS). 


(3) Keep Lt. M. Fleming advised of students progress. 


Hee, 





c. U. S. Army Human Engineering Laboratory, Pacific 
Missile Test Center (PMTC): 
(1) Provide required number of copies of all course 
materials for distribution to subject students. 


(2) Provide point of contact to Lt. Fleming. 


(3) Provide required plans, schedules and procedural 


guidelines as indicated in Section IV. 


203. Points of Contact. 
a. Naval Postgraduate School 


Lt. Martha M. Fleming 
Evaluation Director 

Naval Postgraduate School 
SMC # 1340 

Monterey, California 93940 
Autovon: 878-2536 
Telephones: 408-646-2536 


Cdr. Bill Moroney/ Dr. R. A. McGonigal 
Thesis Advisor 

Naval Postgraduate School 

Code 55 MP 

Monterey, Califoria 93940 

Autovon: 878-2620/2594 

Telephone: 408-646-2620/2594 


Professor Richard S. Elster 
Secondary Thesis Advisor 
Naval Postgraduate School 
Code 54Ea 

Monterey, California 93940 
Autovon: 878-2792 
Telephone: 408-646-2792 


b. COMOPTEVFOR 


Lt. Vickie Bonnano 
Command Course Supervisor 
COMOPTEVFOR, Code O2A 
Naval Station 

Norfolk, Virginia 23511 
Autovon: 690-5598 
Telephone: 804-444-5598 
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DEPCOMOPTEVFORPAC 


Capt. Robert E. Sheridan 
Command Course Supervisor 
DEPCOMOPTEVFORPAC 

NAS North Island 

San Diego, California 92135 
Autovon: 951-6970 
Commercial: 


AFT&E Center/TELH 


LCOL P. A. Crowley 

Command Course Supervisor 
AFT&E Center/TELH 

Kirkland AFB, New Mexico 87117 
Autovon: 244-9606 

Commercial: 


TESTG /ENAH 


Cyrus T. Crites 

Command Course Supervisor 
2620 TESTG/ENAH Stop 239 
Edwards AFB, Califormia 
Autovon: 350-3334 
Commercial: 


Pacific Missile Test Center, U. S. Army Human 
Engineering Laboratory 


Cdr. Tom Jones 

Pacific Missile Test Center 
Code 1226 

Pt. Mugu, California 93042 
Autovon: 351-8981 
Commercial: 805-982-8981 


ASD/ENECH 


Dr. Richard Shiffler 

ASD/ENECH 

Wright Patterson AFB, Ohio 45433 
Autovon: 785-6010 

Commercial: 


Capt. Don Loose 

Electronics Systems Division 
Hanscom AFB, Massachusetts 01731 
Autovon: 478-2825 

Commercial: 
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Section III 


Scope of the Evaluation 


301. Objectives. In accordance with the recommendations of 
reference (a), standardized objectives upon which this course 
is to be evaluated will follow the procedures found in refer- 
ences (c) through (j) and as follows: 

a. Course Objectives. The course objectives must be 
developed according to accepted standardized structure in order 
to prepare the student to perform his/her job in testing and 
evaluating human factors. 

b. Course Material Design. The format of the course 
materials must aide the student in accomplishment of the stated 
learning objectives. The following areas are included in this 
evaluation: 

1. Sequence and structure 
2. Figures and Tables 

3. Symbology and legends 
4. Layout and format 

c. Course Content. The course content must support the 
course learning objectives. 

d. Course Presentation. The design of the course pre- 
sentation aids the student in accomplishment of the course 
learning objectives without the aid of an on-site instructor. 

In addition to the above, future plans for the following 
will be reported: 

a. Course Avallablility. The course materials should be 
maintained in sufficient quantity to meet the needs of those it 


will serve. 
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b. Course Maintainability. The course materials must be 
in accordance with the most recent version of the references 
upon which it is based and human factors design developments. 

c. Course Supportability. Assistance must be available 
by correspondence with the issuing agency. Feedback must be 
provided to the student on his/her performance. The life cycle 
cost of the course must be available for future planning and 


assessment. 


302. Criteria. The criteria below are in accordance with ref- 
erences (e) through (i) and other resource materials listed in 
the bibliography. 

a. Qourse Objectives. The course learning objectives 
should have all the characteristics listed below. 

1. Objectives must be a statement of student behavior 
(action), such as the creation of a product or some other overt 
act, which can be accepted as evidence that the intended out- 
come has occurred. 

2. The behavior must describe specifically all out- 
comes that will demonstrate that learning has occurred. 

3. The student behavior called for must be capable 
of observation and evaluation within the learning and testing 
environments. 

4%. The objective must be stated in learner rather than 
teacher terms, i.e., actions which the student will perform 
rather than what the instructional materials will “say or do". 


5. There must be a standard against which the student 
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pehavior will be measured. It must be fully specified. 
6. The statement of the conditions under which the 


student behavior will occur must be fully specified. 


Additionally, students must be able to satisfy the ob- 
jectives during the normal performance of their jobs in human 
factors test and evaluation. 

b. Course Material Design. 

1. Sequence and structure. The learning objectives 
should be arranged in the sequence in which instruction will be 
presented to the student. 

2. Figures and Tables. Sufficient information should be 
provided with each figure and table to allow the student to 
apply it as directed within the sourse materials and during 
performance of his/her job when left only with the student 
supplement. 

3. Symbology and Legends. The symbols and legends 
presented in the course materials should be implicit 
and standardized where possible. 

4%, Layout and format. The layout and format must aide 
the student in accomplishment of course objectives and encourage 
motivation and desire to complete the course. 

c. Course Content. The mean score of the course groups’ 
second diagnostic tests must be at least one standard deviation 
above the mean score of the control groups’ second diagnostic test. 

dad. Course Presentation. The instructions and verbal context 
of the course materials must allow accomplishment of course 
objectives and course completion without the aide of an on-site 


instructor. 
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303. Evaluation. 

a. Site selection. The evaluation of the HFE course will 
be conducted at various evaluation sites which are involved in 
test and evaluation of systems involved in the acquisition 
cycle. Testing at these sites will provide a realistic environ- 
ment in which to exercise the course. The course is designed 
for personnel with this same job type and responsibilities. 

b. Personnel selection. Subjects participating in this 
evaluation will be drawn from three military commands whose 
primary mission is operational test and evaluation. Some per- 
sonnel will be volunteers, while others may be assigned to 
participate. 

l. Specialty area. It is anticipated that the 
evaluation subjects will be drawn from the various divisions 
within each command (air, surface, subsurface and special war- 
fare). 

2. Special requirements. No constraints regarding 
rate, rank, AFS, grade, educational level or prior experience 
will be placed on participants. The only requirement is that 
each subject be actively involved in planning and performing 
operational test and evaluation. 

c. Data Collection. Data sheets, course diagnostic 
tests and questionnaires for use in this evaluation are con- 
tained in Annex A. Copies will be distributed to Command 
Course Supervisors prior to beginning the evaluation. More 
specifics regarding the actual evaluation procedures are pro- 


vided in Section IV. 





304. Limitations to Scope. While the HFE Self-Instruction Course 
consists of 40 lessons, only the first 20 are scheduled to be 
evaluated. All activities taking part will be provided with 

all 40 lessons in order to provide subjects the opportunity to 
complete the course at a pace faster than that recommended by 


Pacific Missile Test Center. 


The criteria established for assessing the adequacy of the course 
objectives require a prolonged period for a full and complete 
evaluation. This evaluation lasts only for a period of 30 

days and will only analyze the completed questionnaires of the 
participating subjects. This will only provide a preliminary and 
limited assessment. Therefore, it is recommended that a second 
questionnaire be drafted and distributed to the course par- 
ticipants six months after course completion. The subjects then 
will have had an opportunity to apply what the course taught 


them and make a more accurate assessment of of its value. 
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Section IV 


Evaluation Guideline 


401. Evaluation participants and HFE Course Materials. The 


HFE course is designed to provide Human Factors training to 
personnel responsible for planning and conducting testing in 

human factors. This is part of the job of an Operational Test 
Director (OTD) as stated in references (c) and (j). The 

personnel aiding in the evaluation of this course should therefore 
have the same responsibilities. This is the reason for the 
selection of the particular evaluation sites. This course 

was designed for the job and 1s not limited as to rate, rank, 

Air Force Specialty (AFS), grade, educational level or prior 


experience of the participant. 


402. Evaluation Procedures. The HFE course will be conducted 
as follows: | 
a. Student Information Forms (SIF's) must be filled out 

by the CCS for each course participant (See Annex A). -@t thas 
time, each student will be assigned a 4-digit code. The first 
two digits (from the left) identify each individual command: 

COMOPTEVFOR O1LXX 

DEPCOMOPTEVFOR O2XX 

AFT&E Center O3XKX 

TESTG/ENAH O4XX 
The last two digits are then assigned in numerical sequence to 
each participating student: 

Student 1 XXO1L 

Student 2 XXO02 
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Examples COMOPTEVFOR -- 3 students 


Student 1 0101 
Student 2 0102 
Student 3 0103 


This HFE SIF course number should be retained by both student 
and supervisor. It should appear on all correspondence, tests, 
questionnaires and SIF‘s of each student. 

b. After an SIF has been completed and prior to beginning 
the course, each student will be given a diagnostic test, to 
determine the individual level of each student before beginning 
the course. Each CCS will administer the test and make the pur- 


pose of the test clear to each participant. 


In addition, the student must be instructed not to guess if an 
answer to a particular question is not known -- the answer 
should be left blank. The tests will be mailed back to NPS 


immediately upon completion for retention and evaluation. 


c. Following the diagnostic test, each student will be 
provided with the course materials and references. A schedule 
for timely completion of the course will also be provided. Upon 
completion of the course, a final diagnostic test will be ad- 
ministered to each student, by the CCS. This test is also 
designed around the course objectives. While the questions are 
not the same as the first diagnostic test, they will basically 
cover the same material. This test is to be mailed back to NPS for 
scoring. The performance of the student on each of the diagnos- 


tic tests will then be compared. Should the participant wish to 
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to know his/her score on the test(s), it/they will be provided 


by the Evaluation Director at NPS upon request. 


d. A questionnaire will be provided to each student who 
begins the course to provide feedback on individual opinions 


about the course after each has completed it. 
e. A questionnaire will also be provided to each CCS. 


I. Questions asked by the student should be recorded 
by the CCS in the provided log. Each question should also be 


identified with an SIF number. 





403. Test 01, Course Objectives. 
a. Object. To assess the extent to which the course 


objectives support the tasks required of a human factors test 
and evaluation agent. 

ob. Procedure. A task analysis of the human factors 
specialist will be compared to the course objectives. Ques- 
tionnaires regarding the course objectives will be distribu- 
ted to various resources dealing with human factors test and 
evaluation and curriculum design, 

c. Data Requirements. <A task analysis of a human 
factors test and evaluation specialist and completed question- 
naires are required. 

d. Data Analysis. Course objectives will be compared 
to the task analysis to evaluate how adequately the course 
objectives support the person responsible for performing 


human factors test and evaluation. 


404, Test 02. Course Materials Format. 

a. Qbject. To assess readibility, standardization and 
clarity of format according to references (e) through (i). 

b. Procedure. Questions directed to the CCS by each 
participant relative to the HFE course for clarification or 
explanation will be recorded. Questionnaires will be provi- 
ded to students and CCSs upon course completion and/or end of 
test period, which ever applies. 

ec. Data Requirements. All recorded student questions, 


questionnaires and comment sheets will be forwarded to the 
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to the Evaluation Director at NPS, upon completion or end of 
test period which ever is applicable. 

d. Data Analysis. Questionnaire responses will be 
evaluated and the results tabulated. (See Annex B for further 


details.) 


405. Test 03. Course Content. 

a. Object. To assess the extent to which the course 
content suoports the course objectives. 

b. Procedure. Two diagnostic tests will be administer- 
ed to each student; one prior to beginning the course and one 
following its completion. A questionnaire will also be provi- 
ded at course completion. The SIF number will be placed on 
each. 

ec. Data Requirements. Participant test answer sheets 
and completed questionnaires must be returned to the evalua- 
tion director at NPS as they are turned in to each supervisor. 

d. Data Analysis. A mean and its standard deviation 
will be computed for each of the diagnostic test scores. The 
mean scores will then be compared. The individual responses 
to test questions will also be compared. 

Both diagnostic tests will also be administered to 
a control grouv. These individuals will not ve taking the 
course. The purpose will be to substantiate the reliability 
of the tests, and to assess whether the scores of the course 


participants change more than do scores of the control group. 
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Individual questionnaires will also be evaluated and 


analyzed. (See Annex B for further details). 


406. Test 04 Course Presentation. 

a. Object. To assess the clarity of the presentation 
of course content, in order to allow the student to complete 
the course without the aid of an on-site instructor. 


bd. Procedure. See Test 02. 


c. Data Requirements. See Test 02. 
d. Data Analysis. See Test 02. 


407. Test 05. Course Availability. 
a. QObdject. To report on the plans for making course 


materials available to future students. (Implementation and 
Control). 

db. Procedure. Review the plans PMTC would recommend a 
for making course materials available for general use. 

c. Data Requirements. Provision of the implementation 
and controls plans from the issuing agency will be required. 

ad. Data Analysis. The response of the issuing agency 


will be reported. No other analysis is intended. 


408. Test 06. Course Maintainability. 
a. QObject. To report on the plans for keeping the 


instruction manuals up to date with the applicable military 
standards it references and latest human factors developments 
and procedures. 


b. Procedure. Review the plans of the issuing agency 











for scheduling updates in order to keep the course content 
current. 

c. Data Requirements. Provision of the course main- 
tenance plans will be required of the issuing agency. 

d. Data Analysis. The response of the issuing agency 


will be reported. No other analysis is intended. 


OO. Test 07. Course Supportability. 

a. Object. To assess the plans and procedures for 
providing adequate support to students taking the HFE Self- 
Paced Course. 

b. Procedure. Review the plans and procedures provid- 
ed by the issuing agency for supplying the support required 
by the student for successful completion of the course. 

c. Data Requirements. The issuing agency will pro- 
vide the required documents to the NPS Evaluation Director 


for review. 


d. Data Analysis. See Test 06. 





Section 5 
Report(s) 


501. General. The results of this HFE Self-Paced Course 
Evaluation will be provided in the Evaluation Director's 
thesis. 





ANNEX 8 


CCS LOG FORMAT 
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INTRODUCTION 
FOR 
COMMAND COURSE SUPERVISOR (CCS) 


You have been identified by your command to be my point 
of contact with your command in the evaluation of the Human 
Factors Engineering Self-Instruction Course during the period 
1 September - 31 October 1981. The actual length of the 
evaluation at your command may vary depending on how quickly 


the individual student participants complete the course. 


In reading the CCS Guidelines, you will find that your 
responsibilities, very basically, consist of: 
(1) Filling our Student Information Forms (SIF) 
a. asSigning SIF identification numbers. 
(2) Distributing and collecting diagnostic tests/ 
questionnaires, comments and course materials. 
(3) Maintaining your CCS log 
a. Student Progress Record 
b. Student Question and Comment Record 
c. Student Questionnaires 
d. CCS Questionnaires 
e. Evaluation Completion Checklist 
(4) Contacting me for the answers to any questions 
which the student participant may ask that are not obvious 
to you or have not been provided in the material sent 


you. My phone number is: Autovon: 878-2536 
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(5) Returning all materials and correspondence as 
each is completed; but no ter than 1 Novemb 
Mail tos 
Lt. Martha M. Fleming 
Naval Postgraduate School 


SMC # 1340 
Monterey, Cal. 93940 
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EVALUATION GUIDELINES 
FOR THE 


COMMAND COURSE SUPERVISOR (CCS) 


1. Student Information Forms (SIF) and Diagnostic Test #l: 
It is recommended that you arrange to meet with participants in 
a group. 
a. Before the meeting, it is essential that you: 
(1) Read the SIF Instruction sheet to be sure that you 
understand what is wanted in each question. 


(2) Fill in the SIF numbers on each forn. 


b. During the meeting: 

(1) Explain what the evaluation is all about. Hand out E0-1/4. 

(2) Hand out the “Privacy Act Statement” (DA Form 4368-R, 
1 May 1975), which is attached to the front of the SIF. This 
form basically promises them confidentiality of the information 
they provide in the SIF and the questionnaires. As long as they 
follow directions provided, only the participant and the Eval- 
uation Director at the Naval Postgraduate School will see their 
responses to questionnaires and test results. As it states at 
the bottom of the form, they may keep DA Form 4368-2, but do 
need to return the attached SIF. 

(3) SIF: Point out that “name” is optional. The rest 
should be self-explanatory. If not, you have the SIF Instruction 
Sheet which should provide you with any necessary clarifying 


information. 
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(4) Diagnostic Test Number 1: After the SIF has 
been collected: 

(a) Pass out the "Diagnostic Test Number 1", 

(bo) Read through the instructions with the 
students orally. Be sure to highlight those statements or 
words which are underlined. 

(c) Offer them the opportunity to take the 
test in a room where it is nice and quiet; where they won't 
be disturbed. They may take the test back to their desks or 
spaces, which ever they would feel most comfortable doing. 
Even though the test is designed to see how much they know 
about the world of human factors before taking the course, 
the test is not going to be graded, per se as there is no a 
standard against which to measure. The students are ON THEIR 
HONOR to do the test by themselves without reference material 
or other help. The test must be turned in by the end of the 
working day. 

(d) Should some choose to take the test then 
and there, position yourself in the room so that you are avail- 
able to answer _ and record questions the participants ask of 
you. Do NOT walk up and down the aisles. This is in an 
effort to relax the tensions many people have when they sit 
down to take any kind of a test. 

(e) Remember, there is no time limit, so 
allow yourself adequate time for proctoring the test. For 


planning purposes, the test is designed so that even with no 
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previous background in human factors, the test should be 
easily completed within 45 minutes. Students, however, are 


not required to complete the test within this time period. 


2. Command Course Supervisor Log: The CCS Log consists of 
the following: 
SIFs for all participants 
Student Progress Records 
Student/CCS Questions and Comments Record 
Unanswered Student Questionnaires 
CCS Questionnaire 
Evaluation Completion Checklist 
FOR FURTHER CLARIFICATION: 

a. SIF: Keep these in the appropraiate section of your 
log. Dividers have been provided. See also the "SIF Instruc- 
tion" sheet. 

b. Student Progress Records: Record the completion date 
for: 

(1) SIFs 
(2) Diagnostic (D-Test) Test #1 
(3) Each lesson book 
(4) Diagnostic Test #2 
(5) Student Questionnaire 
(6) CCS Questionnaire 
Be sure to keep track of the proper SIF number for each student. 


c. Student Questions and Comments Record: Each time 


either you or a student have a question or comment, record 





~~ eae — 


= —_— 


the following: 

(1) date. 

(2) SIF number. 

(3) the question or comment. 

(4) the answer provided. 

ad. Student Questionnaires: Blank questionnaires have 

been provided so that they are available as each student 
completes the second diagnostic test. These may be taken 


away for completion, 
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EVALUATION OUTLINE 


I. Purpose of the Evaluation: The purpose of this evaluation 
is to assess the operational effectiveness of the Human Factors 
Engineering (HFE) Self-Instruction Course and its potential 


value for use in the Fleet test and evaluation community. 


II. Course Description: This course is divided into three 
major sections, Section I (lessons 1-5) deals with the human's 
capabilities and limitations. In this section, the history 
and continued need for human factors engineering will be in- 
vestigated. Lessons 6-10 are concerned with the physiological 
capabilities and limitations of human beings. Lessons 11-13 
discuss how the proper design of controls and displays makes 
use of the information learned in the previous lessons. Lessons 
14-19 take a look at the human being as he/she interacts with 
his/her environment, followed by a review of the first section 
in lesson 20. 

The primary focus of Section II is upon the human as he/she 
fits into the system. Lessons 21-27, investigate the role of 
the human engineer in various aspects of analysis, such as 
systems analysis, cost analysis, task analysis, etc. Lessons 28-29 
are concerned with the selection and training of personnel. In 
lessons 30-34, the student will receive a short course on 
experimental techniques and statistical concepts. Finally, 
lesson 35, will offer a review of Section II. 

Section III is entitled “Human Factors in the Military”. In 
this section, lessons 36 and 37 focus on human factors organ- 
izations, documentation, and future application. 
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In order to give the students a practical application of 
what has been presented in the course, lessons 38 and 39 will 
ask him/her to work on a ‘real world' problem. Finally, in 
lesson 40, a more typical review of the entire course is pre- 


sented. 


III. Purpose of the HFE Course: The HFE Self-Instruction Course 
was developed to satisfy a need for increased awareness and more 
indepth understanding of Human Factors. 

The Comptroller General, in his report to the Congress of 
the U. S., dated January 29, 1981, (PSAD-81-17), entitled 
Effectivenss of U. 5S. Forces Can Be Increased Through Improved 
Weapon System Design, highlights this need and identifies a 
deficiency in the performance of various weapons systems "be- 
cause the Department of Defense does not pay enough attention 
to logistic support, human factors and quality assurance during 
the design phase of the acquisition process. These problems 
deter the system's effectivenss to defend our country in case 
of war. 

GAO therefore makes recommendations to improve the manage- 
ment and planning of ownership considerations that have an 
impact on the effectiveness of a weapon system.” 

The HFE Self-Instruction course was developed in order to 
provide a basic knowledge of human factors and how to test for 
its effectiveness. The course has not been evaluated or used 


on a trial basis in any portion of the Navy prior to this time. 
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IV. Summary of Evaluation Steps: It is anticipated that each 
of the 40 lessons will require on hour completion time. while 
the course developers recommends 40 working days to complete the 
entire course, time constraints levied necessitate completion 
prior to 31 October 1981. 
(1) Introduction by Command Course Supervisor (CCS) 
(2) Receive Privacy Act Statement 
(3) Fill out Student Information Form (SIF) 
(4) Receive instructions for and take Diagnostic Test #1 
a. student supplement 
b. lesson booklet 1-5 
c. applicable references 
(6) Return each lesson booklet as it is completed and pick 
up the next one in the series, until all 40 lessons are completed. 
(7) Receive and take diagnostic test number 2 
a. return test in sealed envelope 
(8) Fill out student questionnaire 
a. Indicate whether or not test results are desired 
b. Be sure SIF number is visible on envelope 
ec. Return questionnaire to CCS in sealed envelope 
within two working days or not later than 31 October 1981, which 
1s soonest. 
Should you experience any repeated difficulty or delays in 
locating your CCS, feel free to contact me. I do not have my own 
phone, however, a secretary will take the message and I will 


get back to you. Please remember that there is a three hour 


167 5e=3 





time difference between the East and West Coasts. Your 1345 is 


our 1645 and the secretaries leave at 1630. 


Lt. Martha M. Fleming 

Naval Postgraduate School 
SMC #1340 

Monterey, California 93940 


Autovon: 878-2536 
Commercial: 408-646-2536 
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DIAGNOSTIC TESTS INSTRUCTIONS 


Purpose: The purpose of the first diagnostic test is simply to 
see how much you know about human factors before you begin the 
Human Factors Engineering (HFE) Self-Instruction Course. This 
is why we call it a diagnostic test. After you have completed 
the HFE course, a second diagnostic test will be given. The 
purpose of this test is to see how successfully the HFE course 
has increased your awareness of the various facets of human 


factors. 


Guidelines: In order to perform an accurate comparison analysis of 
the two tests, there are several things which we must ask of you 
while taking these diagnostic tests: 

(1) Answer the questions to the best of your knowledge, 
do not get anyone to help you or explain the question to you. On 
the first diagnostic test, you are not expected to know the 
answers. We hope that the HFE course will help you answer the 
questions on the second test. 

(2) Do not guess ! Please, if you understand the question, 
but do not kmow the answer, select the option which indicates 
that you don't know. We realize that this is against your human 
nature and therefore is difficult to do, but we ask that you try. 

(3) This is an evaluation of the HFE Course, not of the 
course participants. You do not flunk or pass; the course does. 
if it is so indicated by your tests, questions, comments and 
questionnaires, the course will return to Pacific Missile Test 


Center for rework. Therefore everything you have to say about 





the course will help us make sure it will be a good one when it 
"hits the streets”. 

(4) If a question does not make sense and needs clarification, 
see your Command Course Supervisor (CCS). 

(5) Your CCS will be recording your questions as you 
ask them. If you have a question, then obviously the materials 
didn't make it clear enough and need to be changed. You can help 
your CCS by writing your question down -- don't forget to put 
your SIF identification number on it. 

(6) Since the HFE course is designed to fit the needs 
of the job of testing and evaluating human factors, rate, rank, 
grade or educational level should not make a difference. Again, 
if it does, the course materials need to be improved. 

(7) Where you take the test is up to you. Your CCS will 
provide you time and a place to take the test, giving you easy 
access to him/her for questions and a place where you will not 
be disturbed. You may elect to take the test back to your desk 
or office, however, remember not only the HONOR SYSTEM of para- 
graph number one, but also that the test must be returmed to 
the CCS in a sealed envelope by the end of that working day. 

(8) Should circumstances beyond your control prevent you 
from finishing the course materials, you still must: 


a. Notify your CCS 


b. Take diagnostic test #2 
c. Fill out the student questionnaire 


Since each test is progressively designed you will still provide 


valuable data by doing (b) and (c) above. 
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(9) Do you want to know your test results? You may 
make that choice while answering the student questionnaire. Your 
CCS will not have the results as the tests are analyzed by the 
Evaluation Director at the Naval Postgraduate School. They 
will be forwarded to you at the end of the evaluation upon 
receipt of your request. 

(10) Do you have any questions? If so address them to 


your CCS now. 


In Summary: 
(1) Do the best you can, 


(2) Do not guess. 

(3) The course is evaluated, not the participant. 

(4) Questions and comments will be recorded in 
logs maintained by your command's Course Supervisor. 

(5) Rate, rank, grade, AFS, educational level and 
prior experience should not make a difference, 

(6) Take your Diagnostic Test #1 where you want, but 
turn it in by the end of that working day. 

(7) Once you start the course, plan on taking diagnostic 
test #2 and filling out the student questionnaire. Return the 
test by the end of that working day and the questionnaire within 
two working days. 

(8) Want your test results? Tell us so in the questionnaire. 

(9) Questions? Ask the CCS. 

F*THANK YOU for participating in this evaluation; without you 


it wouldn't be happening. 
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COMMAND COURSE SUPERVISOR (CCS) 


EVALUATION COMPLETION 
CHECKLIST 


1. Administrative Wrap-Up 
a. Be sure that each of the following has been completed: 

(1) SIF identification numbers on all completed items. 

(2) SIF for each participant. 

(3) Student Progress Records. 

(4) Student/CCS Question & Comment Record. 

(5) Diagnostic Tests #1 and #2 from EACH PARTICIPANT 
beginning the Human Factors Engineering Self-Instruction Course. 

(6) Student Questionnaire from EACH PARTICIPANT begin- 
ning the course. 

b. Retain some way to identify students with their SIF 
identification numbers. This will be necessary should any partici- 
pants request their test scores or evaluation results. We do 
not wish to know them, only that you be able to pass on the 


information upon its receipt. 


2. Return the following together in one package by AIR MAIL -- 
FIRST CLASS -= NO LATER THAN 1 NOVEMBER 1981: 
a. CCS Logs and all completed copies of: 
(1) Student Information Forms (SIF) 
(2) Diagnostic Test #l 
(3) Student Progress Record 
(4) Student/CCS Question and Comment Record 
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) 
(6) 
(7) 


Diagnostic Test #2 
Student Questionnaires 


CCS Questionnaire 


3. Return the following together in a separate package by 


PARCEL POST--Bookrate: 


a. All course materials and extra, uncompleted forms and 


questionnaires: 


(1) 
(2) 
(3) 
(4) 
(5) 
(6) 
(7) 
(8) 
(9) 


Student Supplement 

Booklets for lessons 1-40 

Course references (MIL-STDs and TECHMEMOs) 
SIFs 

Diagnostic Tests 1 and 2 

Student Progress Records 

Student/CCS Question and Comment Records 
Student Questionnaires 


CCS Questionnaires 


ies ECC -+2 





STUDENT PROGRESS RECORD 
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Completion Date (month & day) 


re) 
s9 
® 
eH 
+2 
a 
® 
a 
@o 
bea 
ort 
w 
S 
fr, 
= 
WY 








STUDENT QUESTION & COMMENT RECORD 


DATE | SIF # Student's Question/Comment 

















DIAGNOSTIC PRETEST INSTRUCTIONS 


PURPOSE. The purpose of this test is merely to identify the level of 
your familiarity with human factors engineering BEFORE you take the 
self-paced course. A second diagnostic test will be given after les- 
son 20. If the course fulfills its purpose, the level of your human 
factors engineering familiarity should increase. The results of the 


second test should support this theory. 


FORMAT. This prediagnostic test has eight pages and 28 questions. If 
the test is designed correctly, it should require less than one hour to 


ciemp lete. 


DESIGN. There are several different types of questions: 

1. Multiple Choice - These questions are identified by such terms 
as "CHOOSE" or "SELECT". One or more answers may be correct. Identify 
your choice by placing an "X" in the space provided. 

©. Matching _ The word "MATCH" denotes such questions. Only one 
answer should be chosen for each term on the left. Some of the phrases 
or acronyms on the right may be left unused; don't worry, the question was 
designed that way. 

3. Fill in the Blank - Words such as "NAME", "IS CALLED" or "LABEL" 
identify these types of questions. Sentences requiring completion are also 
examples of "fill ins". You should not need to use more than 5 words to 
answer those questions. 

4, Short Answer - These questions are identified by such terms as 


"EXPLAIN", "DESCRIBE", "DEFINE", "WHY", "WHAT, and/or ask for examples. 








Be brief and to the point, but remember, that someone else must be able to 
understand your answer; so work on its clarity. 

5. Application - One question asks you to redesign, if necessary, 
a display mechanism. Should you decide it is needed, DRAW the redesigned 
mechanism in the space provided. Remember, however, you may be happy with 


the displays as they are presented. Should that be the case, do nothing. 


TEST VALIDITY. It is essential that you do this test by yourself, without 
the use of references or outside resources. To do so will affect the 


validity of the analysis method selected for this evaluation. 


GUESSING. If you want to guess, go ahead. You would anyway, even if I 


told you not to do so. 


GRADING. Keep in mind that the test is being graded, not you. Should 
you like to know the score the test made, you will have the opportunity 


to let us know at the end of the evaluation period. 
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Diagn o nC. PRETEST 


MULTIPLE CHOICE: Choose the best answer(s). One or more than one answer 
may be correct. Place an 'X' by your choice(s). 


A. The overall objective in taking a human factors engineering course is: 


1. to obtain the human factors engineering background 
necessary to do your job well. 


2. to understand a person's specific capabilities and 
limitations. 


3. to understand the precise military procedures involved 
in a human factors program. 


4. to understand the importance of applying the concepts and 
priorities of human factors engineering in your job. 


B. The main reason for poor system performance is human error which is 
caused by: 


1. inadequate consideration of human performance capabilities, 
skill limitations and response tendencies. 


2. little standardization of ‘controls’. (knobs, levers, etc.). 


3. the fast developing pace of new technologies with which 
the human's capabilities cannot keep up. 


4. people with too little education and too few skills being 
brought into the armed services. 


5. inadequate human factors input into system man-machine 


interface. 


C. There are three main areas of the human's information processing capabilities. 
MATCH the terms on the left with the appropriate example(s) on the right. 


1. selective attention a. hear a signal above 20,000 CPS 
2. physiological processing b. interrupting normal procedures 
limitations to react to an alarm bell. 

3. channel capacities c. listening to four incoming 


ship to ship and two secure 
voice radio messages simultan- 
eously on the bridge of a ship 


d. breaking out in a cold sweat 


when required to perform a 
specific duty. 





& fe 


D. MATCH the sensory modalities with the related stimulus dimensions: 


i vision 1. saltiness 
2. audition 2. hue 

3. odor 3. pure tones 
4. taste 4. smell 


E. EXPLAIN WHY each of the following areas is important in human factors 
design evaluation: 


1. information processing capabilities: 


2. sensory modalities: 


F. The science dealing with measurement of the physical features and functions of 


the body is called : 

G. The term which refers to the measurement of human body dimensions in a fixed 
position is ; 

H. NAME the term which indicates that body dimensions are determined from 


body positions which occur with movement. 


I. EXPLAIN WHY body dimensions are an important human factors consideration. 
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J. LIST TWO examples which support your definition on the importance of 
body dimensions as a human factors consideration: 


i 


K. NAME five environmental considerations which may affect a human's performance: 
3 


bee 


L. CHOOSE which phrase deals with the principle of anthropometric design: 


1. use of population stereotype 
2. sequence of use analysis 


3. 5th -— 95th percentile humans 


{~ 


design for the extreme individual 
M. CHOOSE which phrase deals with the principle concerned with the general 
design layout. 
___.__i. sequence of use 
2. length of use 
3. vigilance capabilities of the operator 


4. frequency of use 


3. function of a component 








N. NAME FOUR methods of gathering human factors data on human activities: 
ee 
Zz, 
Se 


4. 


O. What is the purpose of link analysis? GIVE AN EXAMPLE. 


P. Knobs should have basic standardized functions. MATCH the terms on the 
right with the figures on the left. 


Column 
(1) (2) Column (1): 
a. fractional reaction 
b. discrete reaction 


c. multiple reaction 


Column (2): 


d. less than 1 full turm and 
position is not very important 


e. less than 1 full turn where 
position is important 


f. at least 1 full turn 


@ 
a) 
_. GB 
eo; 


page 5 


Q. CHOOSE which of the combined effects of temperature and humidity 


the human body will adapt 
1. extreme heat 
2. extreme cold 
3. extreme heat 


4. extreme cold 


R. 


(1) quantitative OR qualitative 


(2) static OR dynamic 





a. aircraft ‘turn and 
bank indicator’ 


CL) 
2) 


to most quickly: 
and high humidity 
and high humidity 
and low humidity 


and low humidity 


LABEL the following display mechanisms as: 


AND 





b. automobile c. submarine depth 
temperature gauge 
control 

Ge) (1) 

(2) 2) 


S. If you would improve the designs presented in the figures above, SHOW what 
the recommended design change would look like in the space provided below: 


183 








page 6 


DEFINE the term noise: 








There are eight concepts for evaluating control design. Control coding 
is one of these. NAME FOUR of the six factors which are used when 
evaluating control coding: 

i. 


2. 


Another of the eight concepts for control design deals with direction of 
control movement. Briefly EXPLAIN WHY this is important: 


When do the greatest performance decrements occur during a watch-standing 
duty? CHOOSE the best answer(s): 


1. first 30 + 60 minutes of the watch 

2. last 30 - 60 minutes of the watch 

3. last 30 - 60 minutes before lunch break 

4. workspace temperature is between 75°F - 82°F 


5. several short rest periods are given between watches 


What is a task analysis? EXPLAIN: 
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Y. Why is a task analysis so very important in the field of human factors 
design and evaluation? 


Z. MATCH the acronyms with the associated terms: 


1. radiation a. PPM 

2. frequency be BOL 

3. atmospheric contamination c. RAD 

4. pressure d. CPS/Hz 
meee) intensity e. CLO 

6. hearing ioss f. dB/PNDB 

7. insulation g. PTS/TTS 


AA. NAME TWO key military standards (MIL STDS), handbooks or U. S. Army 
Human Engineering Laboratory Technical Memorandums which provide guidelines 
and/or criteria for: 


lL. control design 


2. environmental factors/ 
considerations 


3. anthropometric data 
BB. NAME ONE key military standard (MIL STU), handbook or U. S. Army Human 
Engineering Laboratory Technical Memorandum which provides guidelines and/ 
or criteria for: 
1. display design 
2. aural non-detectability 


3. noise limit selection 


4. control color coding 





page 8 


5. vigilance 


6. criticism regarding 
stated criteria gov- 
erning human exposure 
to carbon monoxide (CO) 


You have now completed your first diagnostic test. Don't feel badly if you 





think that you didn’t know many of the answers. You weren't expected to, remen- 
ber? This test has been a brief introduction as to what you can expect to 
learn as you proceed through the HUMAN FACTORS ENGINEERING SELF-INSTRUCTION 
COURSE. The first lesson booklet and the student supplement will be handed 

to you when you turn this test in to your COMMAND COURSE SUPERVISOR. 

Beep LUCK !!I!t!! Hope you enjoy the course and THANK YOU for your partici- 


pation and support. 
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DIAGNOSTIC POST TEST 
INSTRUCTIONS 


PURPOSE. The purpose of this test is to measure your familiarity 
with human factors. Some of you will be taking this AFTER com- 
pleting the Human Factors Engineering Self-Paced Course 

Others of you in the “control group” will be taking this test 
without taking the course. Those of you in the "control group" 
will be providing a baseline from which to measure the “course 


group's" test results. 


FORMAT. This diagnostic post test has 7 pages and 29 questions. 
If the test is designed correctly, it should require approximately 


one hour to complete. 


DESIGN. Eventually this course will prepare you to develop a gen- 
eral human factors evaluation outline. This requires a thorough 


familiarity with the following areas of human factors design and 


evaluation: 
Test objectives Terms 
Criteria Definitions 
Analytical methods Design principles and 
Resource documents concepts 


The content of this test will establish your readiness for course 


completion. 


There are several different types of questions: 
1. Multiple choice. These questions are identified by the 
term "CHOOSz”. None, one or more answers may be correct. identify 


your choice by placing an ‘X' in the space provided. 
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2. Fill-in-the-blank. Words such as "NAME", "LASEL" or "List" 
identify these types of questions. You should not need to use 
more than five words to answer these questions. 

3. Short answer. The questions are identified by such terms 
as “EXPLAIN”, "DESCRIBE", "DEFINE", "WHY", “WHAT” and/or ask for 
examples. Be brief and to the point, but remember, that someone 
else must be able to understand your answer; so work on its clarity. 

4. Application. One question asks you to redesign, if necess- 
ary, a display mechanism. Should you decide it is needed, DRAW 
the redesigned mechanism in the space provided. Remember, how- 
ever, you may be happy with the display as it is presented. 


Should that be the case, do nothing. 


TEST VALIDITY. It is essential that you do this test by yourself 
without the use of references or outside resources. To do other- 
wise will destroy the validity of the analysis method selected 


for this evaluation. 


GUESSING. If you want to guess, go ahead. You would anyway, 


even if I told you not to do so. 


GRADING. Should you like to know the results of the test, you 
will have the opportunity to let me know at the end of the evalu- 


ation period. 


Se, 








eee 


DIAGNOSTIC POST TEST 


A. The first item of interest is the TASK ANALYSIS. 


1. When should a task analysis be done? 


2. By whom should the task analysis be done? 





3. Describe what information the task analysis provides. 





B. SELECTION OF PERSONNEL for participation in each test system 
evaluation is an area of critical importance. 


1. Explain why you agree or disagree with this statement. 


2. Name two resource documents that will aid you in selection 


of test subjects: a. b. 


3. Briefly describe the basic characteristics of a typical 


test subject: 








C. WORKSPACE DESIGN AND ARRANGEMENT is the third area of concern. 
1. Name two resource documents that will aid you in evalu- 
ating the physical layout of a workspace (i.e. cockpit or 


bridge). on b. 


20) 





2. Name the analysis technique recommended for evaluating 


the arrangement of components within a workspace. 


3. Name the aspect of evaluating workspace design which con- 
siders the physical characteristics of intended human 


operators. 


4%. Name the two human body dimensions measured when evalu- 
ating the workspace design and arrangement. 
ae 


b. 


Evaluating CONTROL PANEL DESIGN should be next on your list. 
CONTROLS, DISPLAYS, and AUDIO/VISUAL ALARMS are items which 


require specific attention. 


1. Name two resource documents that will provide valuable 
guidelines for evaluating a control panel. 


a. 


2. Name the analysis method/technique which is recommended 
for determining whether the controls and displays are 


positioned optimally: 


3. List the guidelines recommended when deciding whether to 
use an audio alarm or a visual alarm. (DO NOT name re- 
source documents -- list the actual guidelines. ) 

a. 


Joy" 











List six of the eight CONCEPTS for evaluating CONTROL DESIGN: 


a. 





0. 














Select one of the concepts from question 4 and name four 
factors considered when evaluating this concept. 
concept: a. 
factors: (1) 

Cay 

3 

(4) 


Select another one of the concepts listed in question 4 
and explain how this concept could be critical in preventing 


numan errors. 


concept: a. 





explanation: 


There are three classes of knobs, (1) discrete, (2) fraction- 
al, and (3) multiple reaction. Figure 1 shows four knobs. 
Select one knob for each of the controls displayed on the 
next page and place its corresponding number in the blanks 


provided. 
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HEATER IGNITION SWITCH RADIO FREQUENCY 
OFF 





a. D. Ce 


Choice of knobs: 


“Co (3) FP wy 


Figure l. 


8. If and only if you believe it is necessary to do so, 
redesign the dial in the space provided following recom- 
mended design practices. Put in all major and/or numbered 
markers. Minor markers need be illustrated only between 


the first two major or numbered markers. (Figure 2) 





—- 


ORIGINAL DESIGN NEW DESIGN 
Figure 2. Tachometer used in testing rotary equipment. The 


scale unit is .5 rpm. The operational range is 50 rpm. 
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The INFORMATION PROCESSING CAPABILITY of the human operator 
is a critical aspect of design. Even though all the controls, 
displays and alarms meet their requirements, the operators 
may still experience difficulty in performing their necessary 
tasks. These difficulties may be caused by the limitations 
on human information processing capabilities. There are 
three main areas in which these capabilities may be grouped. 


One of these is PHYSIOLOGICAL PROCESSING LIMITATIONS (PPL). 
Give. an example for PPL and name the other two main areas: 


1. PPL example: 
2. second area: 


3. third area: 


Weapon systems function in real world environments, not in 
vacuums. The operator may therefore encounter a variety of 


ENVIRONMENTAL FACTORS which may effect his/her performance. 


1. Name one resource document which provides guidelines in 
the area or environmental factors and their possible effects 


on human performance. 


2. Name five environmental factors and for each factor ex- 
vlain what effect it could have on human performance: 


factor effect 


There are ENVIRONMENTAL CONSIDERATIONS which need to be eval- 
uated. Three of these are (1) ATMOSPHERIC CONDITIONS, (2) NOISE, 
and (3) WORK/REST SCHEDULES. 


1. Define the term noise: 


2. Give an example of an atmospheric condition which is a 
concerm to and can be affected by a human factors evalu- 


ator: 


3. When do the greatest performance decrements occur during 


a watch-standing duty? 


4. Listed below are UNITS OF MEASURE. In the space provided 


name what each of these measures, i.e.. in./length; lb./ 


weight. 
a. CPS/Hz 
Bb. PSI 
ee GLO 
d. PTts/TTs 
e. RAD 
f. dB/PNDB 
g. PPM 





Choose the best answer(s). 


One or more than one answer may be correct. 


Place an “X" by your choice(s). 


1. The overall objective(s) in taking a human factors engineer- 


ing course is(are): 


ie 


to obtain the human factors engineering background 


to do your job well. 


to understand a specific person's capabilities and 


limitations. 


to understand the precise military procedures in- 


volved in a human factors program. 


to understand the importance of applying the concepts 


and priorities of human factors engineering in your 


jen. 


to be able to answer some questions on this #*lo! test. 


2. The main reason for poor system performance is human error. 


Which (if any) of the following are contributing factors? 


i 


inadequate consideration of human performance capa- 


bilities, skill limitations and response tendencies. 


little standardization of 


‘controls’. 


(knobs, levers, etc.). 


the fast developing pace of new technologies with 


which the human's capabilities cannot keep up. 


people with too little education and too few skills 


being brought into the armed services. 


inadequate human factors input into system man- 


machine interface. 








Peo eomCentei P I 


This evaluation was originally designed to cover the entire 
course, lessons 1-40 over a period of 60 days. Murphy's Law 
was rolling full steam, however, and the evaluation had to be 
cut to 30 days and only 20 lessons. You all are the ones who 
lose out, though, because the last 20 lessons deal with quantify- 


ing human factors evaluations and practicing human factors 


concepts. 


Recently, the Government Accounting Office, the Secretary of De- 
fense and others have pointed out that the military is accepting 
weapons systems that can't fulfill their missions. They say 

that as a result we are not ready to meet our military obligations. 
Human error is pointed out as being one of the prime reasons for 
the unacceptable system performance. They also state that inad- 
equate attention to human factors design during test and evaluation 


has set up the operator to make the error. 


If we, the operational test directors, know what design factors 
can cause the human to make errors, we can help prevent them. 
Perhaps I‘m a bit surrealistic, but I always thought our job was 
to deliver good and safe equipment to our military men and women 


and in doing so help save lives. 


This human factors course may have been an assignment to you -- 
maybe even one you didn’t like. Whether you wanted to take tne 
course or not, I sincerely hope you have learned something. If 
you have, then the course is serving its purpose and I ask you to 


please continue with the other 20 lessons. Some flyer or sailor 


Hee 





will be out there thanking you for it. Who knows, it may even bde 


you. 


The course may not be perfect and it is for that reason the evalu- 
ation is being conducted. The questionnaires you will be given 

is your opportunity to tell me how to make it better. If ask for 
your continued dedication and co-operation while filling out the 


questionnaires. The feedback you provide will be invaluable. 


Once again, thank you for your help and participation. 
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COMMAND COURSE SUPERVISOR 
QUESTIONNAIRE INSTRUCTIONS 


Directions: Based upon your observations and your log 
entries during the last four weeks, complete this form. 
Upon completion, submit to the Evaluation Director at the 
Naval Postgraduate School for analysis. This Evaluation 
Director may ask for your assistance and for clarification 


during the Analysis Phase of this evaluation. 


NAME COMMAND 
NUMBER OF STUDENTS DATE 


SAMPLE QUESTION. 


(3) How frequently did students' questions/comments 


indicate: 
0-40% 41-60% 61-100% 
a. they did not like 
coffee. 


b. dissatisfaction 
with today's weather. 








If half of all students' questions/comments relating to cof- 
fee indicated they didn't like it, then place an 'X' in the 
41-60% column, as shown below. 
If there are nine students’ questions/comments relating to 
today's weather, and three are negative, then place an ‘X' 
in the 0-40% column (3 + 9 = 33.3%), as shown below. 

0-40% 41-60% 61-100% 


a. they did not like x 

coffee. ee ee. ——— 
b. dissatisfaction x 

with today's weather. 











**There are 4 pages to this questionnaire, be sure to answer all. 
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COMMAND COURSE SUPERVISOR 
QUESTIONNAIRE 


1. How frequently did students' questions/comments indicate 
dissatisfaction with the: 


(a) LESSON BOOK MATERIALS: 0-40% 41-60% 61-100% 
(1) information frames 
(2) wrong answer frames 











(3) questions at end of 
information frames 


(4) answer choices at 
end of information 
frames 


(5) correct answers 
(6) course length 
(7) format 

(8) lesson length 


(9) educational level 
of course content 


(10) course content 


(11) usefulness of the 
course 


(12) amount of time 
spent on course 


(13) terminology 


(>) STUDENT SUPPLEMENT: 
(1) supplement in general 


(2) graphs 

(3) charts 

(4) symbology 

(5) legends 

(6) lesson outlines 








ae 


u, 


5. 


How frequently did students' questions/comments indicate 
dissatisfaction with the: 





(c) RESOURCE DOCUMENTS: 0-40% 41-60% 61-100% 
(1) documents (in general) 
(2) 1472¢c 
(3) 14748 —— ore 


(d) PRETEST: 
(1) pretest (in general) 


(e) POST TEST 
(1) post test (in general) 


How frequently did students' questions/comments indicate 
that initial attitudes about human factors were: 

(a) positive 

(b) negative 

(c) neutral 


How frequently did students’ questions/comments indicate 
that final attitudes about human factors were: 


(a) postive 
(db) negative 
(c) neutral 


How frequently did students’ questions/comments indicate: 


(a) an irritation with 
course materials. 


(bd) that the course provided 
nothing new. 


(ec) that human factors train- 
ing was unnecessary. 


(d) that the learning mater- 
ials were more hinder- 
ing than helpful. 


(e) that the course was a= 
a waste of time. 


AO 2 CCSQ-3 








STUDENT INFORMATION 


6. How many students failed to complete 20 lessons? 
7. How many students continued beyond lesson 20? 

8. How many students completed exactly 20 lessons? 
9 


. How many students requested to keep the Student 
Supplement? 


10. How. many students requested to keep the entire 
course? 


ADDITIONAL COMMENTS: 
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PURPOSE: The main purpose of this questionnaire is to ob- 
tain information regarding the objectives, content, design 
and presentation of the Human Factors Engineering (HFE) 
Self-Paced Course. Your answers will help to deter- 
mine what actions must be taken to improve the course and 
the quality of human factors training and education. Your 


honest opinions are, therefore, essential. 


INSTRUCTIONS: I have no need to Know who you are personnally. 
No effort wiil be made to identify you. However, for anal- 
ysis purposes, it is necessary to have your Student Identi- 
fication number (SIF). No one besides myself will see these 
questionnaires and no individual information will be related 


to your command. 


Before you begin, your Command Course Supervisor (CCS) will 
show you an example of an "information frame” and a “wrong 
answer frame”. This will provide clarification of terminol- 


ogy used within this questionnaire. 


Directions for answering the questions are provided below: 


SAMPLE QUESTION 


(3) What types of schools Types of 
schools 
have you attended? Circle pre-school 
elementary 


junler hegh 
Senior high 
4eyr high 


your answer(s). 


rete & 


123 4 #5 


206 





If you attended elementary school, you should circle the 
number 2, as has been done below, since the number 2 
corresponds to elementary scnool. If, in addition, you also 
attended a 4-year high school, you should also circle the 


number 5, as it corresponds to 4-year high school. 
O26 


SAMPLE QUESTION. 


(4) In taking college courses, the college preparatory 
courses taken in your high school were 

very effective 

effective 

borderline 

ineffective 


very ineffective 


If you felt the college preparatory courses you took in 
nigh school did absolutely nothing to help you through 
college, place an ‘X* in front of “very ineffective", as 
shown below: 

_._._sovery effective 

_._—s-« effective 

___._—«—«borderline 

_.. ineffective 

= 4 very ineffective 
If you have any questions, please ask your CCS for assistance. 
The questionnaire must be returned to your CCS within two 
working days. Be sure to double check that you have answered 
all questions. 
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HUMAN FACTORS ENGINEBRING SELF-INSTRUCTION COURSE OBJECTIVES 





Upon completion of the Human Factors Engineering Course, the 


student will demonstrate the following capabilities and knowledge: 


i. 
2. 


An understanding of common terms used in human factors engineering. 


A familiarity with human factors references and an ability 
to use them. 


An awareness of potential sources of technical information on 
major human factors topics. 


An understanding of the goals of human factors engineering ina 
materiel development program. 


An ability to integrate human factors principles in a DOD 
Sponsored program, 


An ability to determine human performance requirements ina 
systems concent. 


An understanding of the kinds of factors and forces which affect 
human performance and an ability to identify and measure them. 


An awareness of the differences between field and laboratory 
measurements. 


An awareness of what “experimental control" measures are necessary 
for any test involving human performance and the effects in 
their absence. 


An understanding of basic statistical techniques, such as 
analysis of variance. 


An ability to calculate human performance reliability. 


An abiiity to formulate performance measures for the dependent 
variables of time and error. 


An ability to analyze human performance data within the context 
of "system effectiveness" and “system reliability." 


An understanding of the major techniques used by human factors 
Specialists during system synthesis, design, and development. 


A familiarity with task analyses. 
An awareness of the relationship between human factors engineering 


and the engineering specialists of reliability, maintainability 
and safety. 


An ability to interpret and apply the standards and specifications 
of the human factors engineering community. 


20:8 = _— 
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A. Course Content and Objectives. 


Refer to the “Course Objectives" sheet on the next page and 


answer the following by circling your answer(s). 


(1) Which of the 17 objectives listed have been supported 
(taught) by the lessons within the first half of the 


Human Factors Engineering Self-Instruction Course’ 


1 a 3 4 5 6 7 8 Oem eS SA 5 66 O77 


(2) Which of the 17 objectives listed describe a skill neces- 


sary to test and evaluate a system's human factors aspects? 


1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 #16 17 


(3) Which of the 17 objectives listed would help you do your job? 


iL fa 3 4 G 6 4 8 PeeroOweli eile gol’. 15 16 17 


B. Course Design 


(1) The sequence in which the various lesson topics were 
presented was 
____—so~ very effective 
effective 
vorderline 
ineffective 


very ineffective 


= 


(2) The way in which each concept was developed was 


very effective 
effective 
borderline 
ineffective 


very ineffective 


(3) Direction were given to read the information presented 
in each frame, answer the question at the end of each frame 
and go to the page referenced by the selected answer. If 

the answer was incorrect, the student was referred dack to 
the previous page to select another answer. If the answer 
was correct, the student proceeded with the next "information 


frame”. This format was 


very effective 
effective 
borderline 
ineffective 


very ineffective 


Comment: 


on 








(4) How frequently was the re-enforcement or reteaching 
provided in the “wrong answer frame" sufficient to choose 


the “correct” answer? 


frequently sometimes seldom never 


always 








(5) How frequently was the answer to the question at the 
end of each “information frame” suggested or provided in 


preceding course material? 


always 
frequently 
sometimes 
seldom 


nMever 


(6) How frequently was the answer to the question at the 
end of each “information frame” suggested or provided in 


course material following the question? 


always 
frequently 
sometimes 
seldom 


ever 


rata | 








(7) The quality of the questions at the end of each "infor- 


mation frame" was 


excellent 


good 





borderline 
bad 


very vad 
(8) The quality of the answer choices for each question at 
the end of each “information frame" was 


excellent 


good 





borderline 
bad 


very bad 


C. Course Presentation 





(9) Participation in the HFE course without the aid of an 


On-site instructor was 


very effective 
effective 
borderline 
ineffective 


very ineffective 


to 
p— 
i) 








(10) How frequently was assistance required of the Command 


Course Supervisor? 


frequently sometimes seldom never 





always 
(11) How frequently were words used which required defining 
by an outside source (dictionary, person, etc.)? 


always frequently sometimes seldom never 


D. Student Supplement 


(12) How effective/ineffective were each of the elements 
listed below in aiding your understanding of the concepts 


presented in the course material? 


Very 
Very In- in- 
effec- Effec- border- effec- effec- 
tive tive line tive tive 
tables a on ae SAE nace >. 
charts : =< — a 
graphs ee eC] 
lllustrations —_ as 
symbology ee eee 
legends - a ae os 
terminology a a 
lesson outlines a ae —_— 
arrangement 
of tables, 
graphs, etc. — ee ae os  —_ 


eS 








(13) How do you feel about the student supplement as a ref- 
erence book and human factors guideline during test and eval- 


uation planning stages? 


extremely useful 

of considerable use 
of use 

not very useful 


of no use 


(14) The information given with figures and tables (charts 


and graphs) to vrovide clarity and meaning to each was 


extremely useful 

of considerable use 
of use 

not very useful 


of no use 


(15) The implied ideas and meanings represented by symbols 


and legends in the student supplement were 


always understood 
frequently understood 
___. sometimes understood 
_... seldom understood 


never understood 





E. Resource Documents 


(16) The HFE course materials made use of the resource 


documents listed below 


very 
very border- diffi- diffi- 

easy easy line Culm Cult 
1472B a 
147483 a 
mere So eeeees—a‘“(i‘“‘OswOSCNWOC! 


(17) Use of the resource documents listed below in planning 


human factors tests for a new system is 


very 
very porder- diffi- diffi- 
easy easy line cult cult 
14728 
14743 ee 
HDBK 759 | a 


F. Human Factors Attitudes 


(18) How do YOU FEEL about including a human factors test 


in the evaluation of a new piece of equipment or system? 


very important 
important 
borderline 
unimportant 


very unimportant 


aS 





G. Miscellaneous 


(19) The adventures of Lt. I. M. Eager were tracked through- 


out the HFE course. The stories about Lt. Eager were 


always effective 
frequently effective 
sometimes effective 
seldom effective 


never effective 


COMMENT : 


(20) How frequently did the factors listed below adversely 
affect your completion of the lessons in the first half of 


the course? 
always frequently sometimes seidom never 


Experience 
Education 
Time 

Jodo load 
Course Length 
Format 
Terminology 
Design 


TAD (temporary 
additional duty) 


(**) Please identify, in the space below, any 


problems you had with the following: 


LESSONS BOOKS: 


STUDENT SUPPLEMENT: 


RESOURCE DOCUMENTS: 


PRETEST: 


specific 





BOs. ITzsT: 


(##%) Would you like to keep a copy of the HFE course? 


yes no 





(*##) Wouid you like to keep a copy of the student supplement? 


yes no 





(**%*) Would you like to know your test scores? 


yes no 








(##4) Would you like to see the final report on the HFE 


Self-Paced Course? 


yes no 








WHAT IS YOUR Student Identification Number? 





COURSE OBJECTIVES QUESTIONNAIRE 


COURSE OBJECTIVES EVALUATION 


NAME 





COMMAND 
POSITION/TITLE 
RESPONSIBILITIES 





EXPERIENCE IN COURSE EVALUATIONS (yrs. ) (mos. ) 
TIME AT THIS COMMAND (yrs. ) (mos. ) 


The six characteristics listed on page one have been 
established by NAVEDTRA 106A as criteria by which in- 
structional learning objectives are to be judged. The 17 
learning objectives listed on pages two and four will, 
therefore, be evaluated according to now completely they 


meet the six criteria as explained on the next two pages. 








Directions: Assess each learning objective according to 
each of the six criteria. Place criteria numbers (1,2,3,4,5, 
and 6) in the spaces to the right of each learning objective, 
according to how completely each objective satisfies each 


of the criteria. Each criteria should be rank ordered from 


@ 
"Not at All" to Completely”. “Y se eS 
i He G <“ 
~ Se Ey a 
SAMPLE: a9 hc Qe Ge 


A. A student will be able to 
correctly solve quadratic 
equations. 


CRITERIA FOR COURSE OBJECTIVES 


1. Objectives must be a statement of student behavior 
(action), such as the creation of a product or some 
other overt act, which can be accepted as evidence that 
the intended outcome has occurred. 


2. The behavior must describe specifically all outcomes 
that will demonstrate that learning has occurred. 


3. The student behavior called for must be capable of 
observation and evaluation within the learning and 
testing environments. 


4. The objective must be stated in learner rather than 
"teacher" terms, i.e., actions which the student will 
perform rather than what the instructional materials 
will “say or do”. 


5. There must be a standard against which the student 
behavior will be measured. It must be fully specified. 


6. The statement of the conditions under which the student 
behavior will occur must de fully specified. 








If you feel that objective "A" fulfills criteria 1,2,3, 
and 4 completely, then place the numbers 1,2,3, and 4 in the 
blank under "Completely" as shown below. If you feel that 
objective "A" does not meet criteria 5 at all, place a "5" 
under "Not at All". If it meets criteria 6 to some extent, 


but needs quite a bit of attention, then place a "6" under 


"To Some Degree", as show below. 





> 
oe 9% on 
ys a oF Ao? 
A. A student will be able to 
correctly solve quadratic 5S 6 


equations. 


If you have any questions or ccmments regarding this 
evaluation, please feel free to contact me or write them 


in the space provided at the end of this evaluation. 


Lt. Martha M. Fleming 

Naval Postgraduate School 
SMC ¥ 1340 

Monterey, California 93940 


Autovon: 878-2536 
Commercial: 408-646-2536 
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LEARNING OBJECTIVES 


An understanding of common terms 
used in human factors engineering. 


A familiarity with human factors 
references and an ability to use 
them. 


An awareness of potential sources 
of technical information on human 
factors topics. 


An understanding of the goals of 
human factors engineering in a 
material development program. 


An ability to integrate numan 
factors principles in a Department 
of Defense sponsored progran. 


An ability to determine human per- 
formance requirements in a systems 
concept. 


An understanding of the kinds of 
factors and forces which affect 
human performance and an ability 
to identify and measure them. 


An awareness of the differences 
between field and laboratory 
measurements. 


An awareness of what “experimental 
control” measures are necessary for 
any test involving human perform- 
ance and the effects in their ab- 
sence. 


An understanding of basic statis- 
tical techniques, such as analysis 
of variance. 


An ability to calculate human per- 
formance reliability. 


An ability to formulate perform- 
ance measures for the dependent 
variables of time and error. 


ee) 
Le 
Ud 








CRITERIA FOR COURSE OBJECTIVES 


Objectives must be a statement of student behavior 
(action), such as the creation of a product or some 
other overt act, which can be accepted as evidence 


that the intended outcome has occurred. 


The behavior must describe svecifically all outcomes 


that will demonstrate that learning has occurred. 


The student behavior called for must be capable of 
observation and evaluation within the learming and 


testing environments. 


The objective must be stated in learner rather than 
“teacher” terms, i.e., actions which the student will 
perform rather than what the instructional materials 


will "say or do”. 


There must be a standard against which the student 


benavior will be measured. It must be fully specified. 


The statement of the conditions under which the student 


behavior will occur must be fully specified. 


NAVEDTRA 106A 
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LEARNING OBJECTIVES - = << 


An ability to analyze human per- 
formance data within the context 
of “system effectiveness" and 
"system reliability". 

An understanding of the major 
techniques used by human factors 
specialists during system synthesis, 
design, and development. 

A familiarity with task analyses. 
An awareness of the relationship 
between human factors engineering 
and the engineering specialists of 
reliability, maintainability and 
safety. 

An ability to interpret and apply 
the standards and specifications 
of the human factors engineering 
community. 


Are these the criteria by which you judge learning objectives? 


(Circle one): YES NO 


If the above answer was "NO", please provide a copy of the 


Criteria you do use.) 
Do you approve of these 6 characteristics being used as 
criteria for judging learning objectives? (Circle one): 


YES NO 


Why? s 


PANIES 








APPENDIX F 
DEMOGRAPHIC AND PRE-POST TEST DATA 
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APPENDIX G 
POST TEST RAW SCORES 
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APPENDIX H 


STUDENT ATTITUDE SURVEY 
(QUESTIONNAIRE) 
DATA 


2S 2 








Table 28 


Record of Student Attitude (Questionnaire) Responses 


A. COURSE CONTENT AND OBJECTIVES 


iis 


Wojeceurves taught in the first 


OG). 7 : 1 2 5 4 
Responses: 20 19 11 16 


Sba. 7: 11 12 13 14 
Responses: 2 3 2 10 


Objectives necessary to do 


Soa. *: 1 Z 3 4 
Responses: 8 8 i fi 
Oba. Ff: 11 12 13 14 


Responses: l1 10 12 5 


eS) 
10 


gusle 


20 lessons: 


Pl 4 7 2 
Item Il % 
6 8 

Test and Evaluation: 


Objectives which would nelkpmyeu do your job: 


Obj. #: ie ests 4 


Reoapenses: 10 12 Mi 5 


Obagi. 7: JC NSS 4 a Wem 
Responses: 8 ie BEES 9 








Table 28 - Continued 


COURSE DESIGN 
Student Responses 


Most Most 
Positive Negative 
5 4 3 2 1 
y L / / i fietan 

mercegquence of les- 

son topics Z 14 4 i 3 24 
2. Concept develop- 

ment di ise ) 2 3 24 

Format 1 I Z 5 5 24 
4. Remediation 1 6 9 3 4 Z5 

. Info provided 

before questions 1 ih 9 Z ii 24 
6. Info provided 

after questions 2 5 14 if Z 24 
7. Question quality 1 ll uC 1 if 24 
8. Answer choice 

quality 1 10 ib 1 Z 24 
COURSE PRESENTATION 
9. Course w/o on-site 

instructor 1 8 7 4 4 24 
mvmeereq, use of CCS eS 6 3 - - oe 
11. Terminology--defs. 

provided 6 9 ii 2 ~ 24 





Table 28 - Continued 


tees le DENT SUPPLEMENT Student Responses 


Most Most 
Positive Negative 
5 4 5 % i 


(ee ee ee Total 


12. Effectiveness of course elements 


Tables 2 rs 6 ° - 25 

Charts Z 14 7 - - 25 

Graphs 2 is 8 - - 2 

Illustrations 2 5 6 - - LES 

Symbology 2 8 2 1 a 

Legends - 14 7 1 - Le 

Terminology - iis 6 - Z aS 

Lesson outlines 3 10 8 : A wS 

Arrangement of 

tables 4G graphs - 2 fi 5 iL 23 

ieewsefulness of Stu- 

dent Supplement as 

a reference = 6 10 i i 24 
14. Student Supplement 

(usefulness § self- 

explanatory )--fig- 

ures § tables - 8 13 3 - 24 
15. Clarity of symbols 

and legends 4 Ve! 7 1 1 24 





Table 28 - Continued 


RESOURCE DOCUMENTS 
Student Responses 


Most Most 
Positive Negative 
5 4 5 Z 1 


/ / i / / tocad 


16. Course made reference documents easier/harder to use: 


1472C Z 3) 7 2 = te, 
14748 Z 7 ° Z = 16 
HDBK 759 2 11 ) 2 = 20 


17. Use of ref documents in planning HFE T §& E made easier/ 


harder: 
Aa - 10 6 3 - 19 
1474B - 8 6 3 = iL 
HBB 59 z Bie 5 5 - 19 
HUMAN FACTORS ATTITUDES 
18. Is HFE T §& E necessary: 
12 10 1 i - 24 
MISCELLANEOUS 
19. Story Line Effectiveness: 
1 3 10 4 6 24 





Table 28 - Continued 
MESCELLANEOUS (cont'd) 


Student Responses 


(Never) (Always) 
Most Most 
Postive Negative 
5 4 5 2 i 
(8 | Total 


20. Factors Adversely affecting completion of lessons: 


Experience 10 5 5 5 - 23 
Education 10 4 5 5 i ZS 
Time Available i 1 9 9 3 2S 
Job Load 1 =o fl 11 5 oe 
Course Length 6 6 5 4 Z (LES 
Format 3 6 5 1 9 24 
Terminology 8 Y 9 5 - 24 
Design 5 4 3 4 5 oe 
Temporary Addi- 
tional Dury 6 3 8 5 2 24 
COURSE DESIRABILITY 
ines No Total 
Students want to keep course materials. 10 14 24 
Students want to keep Supplement only. eS 11 24 
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ON TERMINAL LEARNING OBJECTIVES 
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SAMPLE RECOMMENDATIONS 
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Figure 1 


Meagune Cover Page Format 


MODULE 1.0 


HUMAN CAPABILITIES AND LIMITATIONS 


FOR 
HUMAN FACTORS ENGINEERING 


Course Identification Number 


Prepared by 


Pacific Missile Test Center, Pt. Mugu, California 


25 December 1983 
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Ficume 2 


Module Overview Format 


MODULE OVERVIEW 


MODULE 1.0 


Human Factors Engineering 


HUMAN CAPABILITIES AND LIMITATIONS 


Memtnis module .....cccee 


List of Lesson Topics Included in Module: 


Lesson Topic 1 - Welcome to Human Factors Engineering 
Average Time: 


Lesson Topic 2 - Why Human Factors? 
Average Time: 
Lesson Topic 3 - Tragic Mistakes...and Positive 


Consequences. 
Average Time: 


Lesson Topic 20 - Review 
Average Time: 
Total Module Average Time: 


(Insert any general instructions to students concerning the 


use of this Module Booklet and any of its supporting mater- 
tals.) 
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Recure 3 


Lesson Topic Cover Page Format 


HUMAN FACTORS ENGINEERING 


MODULE NUMBER 1.0 


ESsoON LOPTC. 1.1 


WELCOME TO HUMAN FACTORS ENGINEERING 


25 December 1983 
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Figure 4 


Lesson Topic Overview Format 


MOBULE 1.20 


GESoON TORPTC 1.1 


LESSON TOPIC OVERVIEW 
beESoGhelOPRTC 1..1 
WELCOME TO HUMAN FACTORS ENGINEERING 


In this Lesson Topic 


ipesLearning OBJECTIVES of this Lesson Topic are as 
follows: 


iP, 


(Statement to the student that he or she should review the 
"LIST OF STUDY RESOURCES" and read the Lesson Topic LEARN- 
ING OBJECTIVES before beginning the Lesson Topic.) 
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Figure 5 


List of Study Resources Format 


MODULE 1.0 
SeeON TOPIC 1.1 


DicimOr SiUDY RESOURCES 
WELCOME TO HUMAN FACTORS ENGINEERING 


To learn the material in this LESSON TOPIC, you have the 
option of choosing, according to your experience and 
preferences, any or all of the following study resources. 


Written Lesson Topic presentations in the Module Booklet: 


1. Lesson Topic Summary 

2. Programmed Instruction Form of Lesson Topic 

53. Student Supplement containing supporting 
information, charts, graphs, etc. 

4. Lesson Topic Progress Check 

5 Narrative Form of Lesson Topic 


Additional Materials 


1. Student Response Sheets 
a= ODiwwgmams, charts, graphs 
b. Programmed Instruction Response Sheets 
c. Answer Sheet for use with all tests 
d. Notetaking Sheets 


Enrichment Materials 


1. Additonal Resource Documents from the Student 
Packet 

2. Additonal References from the Naval Supply 
system 


You may use any or all resources listed above, including 
the Learning Course Sponsor, but all materials listed are 
not necessarily required to achieve Lesson Topic Objectives. 
The Progress Check may be taken at any time. 
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Figure 6 


Lesson Topic Summary Format 


MODULE 1.0 
BESSON TOPIC 1.0 


LESSON TOPIC SUMMARY 
Welcome to Human Factors Engineering 


(A condensation of the narrative form of the lesson topic 
is placed here). 


Statement to the student: 


"At this point, you may take the Lesson Topic Progress 
Check. If you answer all self-test items correctly, pro- 
ceed to the next Lesson Topic. If you incorrectly answer 
Only a few of the Progress Check Questions, the Correct 
Answer page will refer you to the appropriate pages, para- 
graphs, or frames so that you can restudy the parts of this 
Besson Topic you are having difficulty with. If you feel 
that you have failed to understand all, or most, of the 
Lesson Topic, select and use another medium of instruction: 
Narrative or consultation with the Learning Course Sponsor, 
until you can answer all self-test items on the Progress 
Gmeck correctly." 
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